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PREFACE 


This  work  was  undertaken  by  Mei  Associates,  Inc.  on  behalf  of  the  Armstrong 
Laboratory,  Human  Resources  Directorate  (AIVHR)  as  part  of  a  larger  study  of  Air  Force 
computer-based  training  (CBT).  This  report  presents  the  findings  of  the  Laboratory  in  its  survey 
of  Air  pOTce  CBT  users. 

This  project  benefitted  from  the  many  helpful  suggestions  from  Dr.  Scott  Newcomb, 
Chief,  Instructional  Design  Branch,  and  Amstrong  Laboratory  Scientists  Oqrtain  William 
Dimitroff  and  Mr.  Dennis  Gettman.  In  addition  to  their  significant  input  regarding  the  data 
collection  instrument,  interview  techniques,  and  data  analysis  procedures,  each  of  these  scientists 
helped  the  authors  focus  their  attention  on  matters  of  importance  to  the  Air  Force.  Mr.  Gettman 
also  contacted  numerous  Air  Force  organizations  suspected  of  using  CBT,  so  that  they  might  be 
included  in  the  survey  and  accompanied  the  research  team  to  several  bases  to  facilitate  collection 
of  the  data. 

Special  thanks  are  in  order  to  Dr.  Juanita  Firestone  of  the  University  of  Texas  at  San 
Antonio,  who  provided  valuable  assistance  to  the  authors  in  survey  methodology.  Dr.  Firestone 
focused  attention  on  what  could  be  done  and  how  to  do  it  properly.  Her  fine  attention  to  the 
details  of  statistical  analysis  helped  improve  the  findings  of  the  survey.  Dr.  Firestone  was 
instrumental  in  guiding  the  research  team  through  the  development  of  the  research  plan.  Her 
guidance  helped  the  researchers  determine  what  analytical  and  statistical  techniques  could  and 
should  be  used  in  analyzing  the  data.  Once  data  were  coded,  she  constantly  provided 
suggestions  as  to  what  andytical  procedures  might  yield  the  most  interesting  results.  Her  review 
of  the  findings  helped  eliminate  any  inconsistencies  which  might  otherwise  have  been  there. 
The  research  team  was  also  aided  by  Robert  "Les"  Caldwell,  who  manipulated  the  SPSS 
database  with  diligence,  and  provided  the  insight  of  an  outside  observer  to  CBT  data.  Ms.  Diarm 
K.  Andy  was  also  helpful  by  applying  her  practical,  computer-based  training  expertise  to  the 
development  of  the  questionnaire.  Diaim  spent  numerous  hours  recalling  potential  problem 
areas  she  had  wimessed  first-hand,  and  formulating  these  into  questions  which  could  objectively 
categorize  the  various  aspects  of  CBT  issues. 
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SUMMARY 


Computer-based  training  (CBT)  is  widely  used  by  the  Air  Force  to  conduct  training. 
Numerous  CBT  technologies  are  being  employed  by  Air  Force  organizations  worldwide  as  tools 
for  training  new  skills,  upgrading  or  re^shing  existing  skills,  and  to  maintain  currency  of 
personnel  in  their  jobs.  Some  of  the  CBT  being  used  has  been  developed  by  contractors,  but 
much  of  the  courseware  has  been  developed  in-house  by  various  Air  Force  organizations.  This 
survey  conducted  under  the  auspices  of  Armstrong  Laboratory  examined  the  procedures  used  by 
the  Air  Force  in  developing  CBT  in-house.  The  ultimate  goal  of  the  research  is  to  provide 
suggestions  to  improve  Air  Force  CBT. 

The  survey  sample  consisted  of  253  participants  from  14  Air  Force  commands  and 
operating  agencies  at  26  bases  and  51  organizations.  The  participants  were  70  officers,  117 
enlisted  and  66  civilians  involved  in  CBT  as  managers,  instructors,  developers  or  computer 
systems  experts. 

Several  major  issues  were  investigated  regarding  planning  for  CBT,  selecting  a  CBT 
system,  developing  CBT  courseware,  and  implementing  CBT,  including : 

o  CBT  Planning 

o  Management  of  CBT  Programs 

o  CBT  Development  Personnel 

o  Effective  Use  of  CBT 

o  CBT  Development  Procedures 

Many  Air  Force  organizations  are  hard  at  work  implementing  CBT.  The  effon  expended 
by  numerous  individuals  and  the  pride  in  their  accomplishment  demonstrated  to  the  research 
team  are  typical  of  their  spirit  to  do  the  best  with  what  they  have.  Although  the  findings  of  this 
survey  concentrate  on  potential  problems  that  these  organizations  face  in  developing  CBT,  one 
should  not  overlook  the  concerted  efforts  taking  place  tluoughout  the  Air  Force  to  make  it  work. 

CBT  Planning 

A  majority  of  the  respondents  reported  performing  the  various  steps  involved  in  planning 
for  CBT;  77.6%  perform*^  a  Training  Ne^  Assessment,  67.1%  conducted  a  JobH’ask 
Analysis,  83.1%  analyzed  Trainee  Characteristics,  64.1%  assessed  Facilities  Requirements,  50% 
conducted  a  Media  Analysis,  and  86.9%  developed  Training  Objectives.  While  these 
percentages  indicate  that  many  organizations  are  following  accepted  practices,  some 
organizations  are  not  conducting  a  complete  fiont-end  analysis  for  CBT.  Many  organizations 
surveyed  appear  to  be  using  existing  materials  as  the  basis  for  CBT  (59.4%).  This  has  led  to 
underutilization  of  the  capabilities  of  the  medium;  77.1%  report  that  the  media  were 
predetermined. 

CBT  can  be  a  costly  medium.  Many  respondents  reported  that  budgets  did  not  accurately 
anticipate  actual  CBT  costs  (49.4%).  Several  sources  of  unexpected  cost  contributed  to  budget 
problems;  55.3%  reported  a  longer  than  planned  development  time,  34.2%  bought  more 
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hardware  than  planned,  and  28.9%  bought  more  software  than  planned.  When  funding  shcntfalls 
occurred,  projects  were  frequendy  either  put  on  hold  (41.4%)  or  reduced  in  sc(^  (38.5%). 
Budget  shortfalls  also  had  an  effect  on  individual  lessons  with  7.7%  reporting  they  developed 
shorter  lessons,  25.6%  programmed  fewer  graphics,  and  23.1%  had  less  interactivity. 

Management  of  CBT  Programs 

CBT  managers  are  tasked  to  work  with  a  complex  medium  with  few  guidance  tools.  A 
signifleant  number  of  program  managers  (39%)  had  no  CBT  experience;  11.7%  had  neither 
experience  nor  training  in  CBT.  Many  respondents  commented  about  CBT  management  during 
follow  up  interviews.  While  their  comments  tended  to  blame  management  fOT  most  problems 
involving  scheduling  and  coordination  of  team  efforts,  survey  data  ate  supportive.  Delays  in  the 
completion  of  project  milestones  indicate  that  CBT  managers  need  spc^c  guidance  in  what 
needs  to  be  done  and  how  to  accomplish  it  Respondents  repeat  stnyboards  were  not  completed 
on  time  (35.1%),  on-line  programming  of  lessons  were  delayed  (44.4%),  and  implementation  of 
CBT  courses  was  delayed  (28.2%).  This  survey  identified  several  issues  related  to  CBT 
management  which  need  further  research. 

CBT  Development  Personnel 

The  experience  profile  of  (TBT  development  personnel  is  similar  to  that  of  managers; 
52.6%  of  instructional  developers  and  53.9%  of  subject  matter  experts  were  inexperienced. 
Many  development  personnel  had  neither  experience  ntxr  trairting  (23.1%  of  instructional 
developers;  27.5%  of  subject  matter  experts).  In  addition  to  inexperience,  personnel  turnover 
over  the  life  of  a  project  was  26.6%.  The  most  significant  result  of  tunxrver  was  increased 
development  time  (72.1%).  This  complicated  management  problems. 

Effective  Use  of  CBT 

Respondents  made  use  of  many  CBT  features  such  as  graphics  (95.2%),  audio  (48.2%), 
and  video  (45.4%),  but  specific  applications  tend  to  reflea  that  tlwy  were  cemverting  existing 
uaining  to  CBT.  Factors  considered  in  selecting  CBT  system  hardware  and  software  reinforce 
this.  Training  requirements  were  considered  by  50.6%  of  respondents  when  selecting  hardware 
and  by  36.6%  when  selecting  software.  In  contrast,  far  mote  respondents  considered  software 
requirements  (72.3%)  when  selecting  hardware,  and  hardware  requirements  (67.7%)  when 
selecting  software.  72.5%  of  respondents  reported  using  the  computer  fw  management 
functions  (CMI).  CMI  was  reported  as  a  factor  in  software  selection  by  53.8%  of  respondents. 
The  inexperience  of  the  development  team  affects  the  underutilization  of  CMI  capabilities. 

CBT  Development  Procedures 

CBT  courseware  review  procedures  appear  to  be  somewhat  inefficient,  in  that  major 
revisions  are  made  late  in  the  development  process.  Although  developers  made  extensive  use  of 
such  standardization  aids  as  format  guides  (77.3%),  sample  lessons  (52.1%),  flowcharts  (77%) 
and  storyboards  (83.2%),  significant  revisions  are  made  even  after  lessons  are  already  on-line 
(55.1%).  Although  93.1%  of  respondents  reported  that  courses  were  successfully  validated,  only 
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2^.9%  attempted  to  relate  the  training  to  job  perfonnance  or  made  use  of  supervisor  evaluations 
(18.4%). 

Recommendations 

While  the  findings  hrmn  this  survey  are  not  conclusive,  they  are  indicative  diat  Air  Force 
CBT  is  in  need  of  improvement  Hum  specific  courses  of  action  to  improve  CBT  are 
recommended  based  on  Ae  findings  of  this  survey: 

o  Further  research  is  needed  on  the  quality  of  Air  Force  CBT.  This  survey  did  not 
examine  the  quality  of  Air  Force  CBT;  instead  it  focused  on  issues  of  CBT 
planning,  selection  and  implementation.  Further  research  is  needed  to  determine 
the  elffect  that  current  CBT  development  practices  have  on  the  quality  of  Air 
Force  (TBT,  i.e.,  its  effectiveness  and  efficiency  as  a  training  medium. 

o  Specific  guidance  and  training  is  needed  for  managers  and  developers  of  CBT. 
Data  from  this  survey  suggest  several  problems  associated  with  CBT 
development  Those  involved  in  CBT  should  receive  clear  guidance  and  also  be 
provid^  training  which  not  only  addresses  the  steps  to  follow  in  CBT 
development  but  also  stresses  the  rationale  for  each  step,  how  to  do  it  typical 
problems  which  might  be  enrauntered,  and  provides  the  managers  and  developers 
with  an  opportunity  to  practice  their  skills  before  being  immersed  into  the  job  of 
developing  CBT. 

o  Air  Force  personnel  capable  of  developing  CBT  need  to  be  identified  for  future 
assignments.  If  the  Air  Force  intends  to  remain  in  the  CBT  business,  a  tinal 
recommendation  to  improve  Air  Force  CBT  is  to  provide  a  means  within  the 
personnel  system  to  identify  personnel  with  CBT  skills  for  future  assignment 
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A  SURVEY  OF  AIR  FORCE 
COMPUTER-BASED  TRAINING  (CBT) 
PLANNING,  SELECTION  AND  IMPLEMENTATION  ISSUES 


1.  INTRODUCTION 

The  purpose  of  this  study  was  to  collect,  document,  and  analyze  data  on  problems 
associated  with  Computer-Based  Training  (CBT)  planning,  selection,  and  implementation  in  the 
Air  Force.  The  term  "CBT*  in  this  report  refers  to  all  training  technologies  which  utilize  a 
personal  computer  as  a  centerpiece.  This  includes  interactive  video  (IVD),  digital  video 
interactive  (DVI),  audio,  and  PC-based  simulations.  It  also  includes  other  acronyms  for 
instructional  uses  of  the  computer,  such  as  (Computer-Assisted  Instruction  (CAI).  Computer- 
Based  Instruction  ((CBI),  (Computer-Assisted  Learning  (CAL),  and  Computer-Managed 
Instruction  (CMI). 

The  use  of  CBT  technology  is  becoming  widespread  throughout  the  Air  Force.  Various 
reports  estimate  there  are  approximately  50-100  new  system  "starts”  annually.  Indications  are 
that  the  scope  of  CBT  technology  in  the  Air  Force  is  currently  quite  extensive  and  is  expanding 
every  year. 

When  (CBT  technology  has  been  properly  planned  for,  and  is  implemented  based  on  a 
sound  plan,  it  can  have  a  positive  effect  on  l^th  training  effectiveness  and  efficiency  (Kemner- 
Richar^on,  Lamos  and  West,  1984;  MacNiven,  1987).  However,  when  CBT  is  not  properly 
planned  for,  when  an  inappropriate  CBT  system  is  selected,  or  when  a  CBT  implementation 
encounters  problems,  the  results  can  be  just  the  opposite.  When  improperly  applied,  CBT  can 
potentially  result  in  ineffective  instruction  leading  to  substandard  learning,  or  increased  costs  due 
to  longer  training  times,  courseware  development  problems,  and  logistics  or  maintenance 
problems.  In  addition,  the  inappropriate  application  of  CBT  in  a  training  environment  can  result 
in  adverse  impacts  on  a  training  organization's  operating  structure,  functioning  and  resources. 
Given  the  current  state  of  plaiming  for  and  selecting  C3T  systems,  these  problems  do  not 
manifest  themselves  until  after  a  CBT  system  has  been  developed  or  implement^ 

In  determining  the  appropriate  media  application  for  their  training  environment,  Air 
Force  and  other  Department  of  Defense  (DoD)  users  need  to  be  able  not  only  to  select  CBT  from 
a  group  of  other  media  alternatives,  but  also  to  select  the  most  appropriate  (TBT  system  for  their 
specific  training  needs.  Planning  for  CBT  technology  is  frequently  the  result  of  the  technology 
driving  the  requirements,  rather  than  using  the  requirements  to  specify  the  technology,  as  for 
example,  when  C3T  technology,  including  hardware  and  software  is  acquired  prior  to  the 
definition  of  user  training  requirements.  This  can  result  in  the  inappropriate  use  of  the 
technology,  i.e.,  forcing  the  requirements  to  match  the  available  technology.  The  selection  of 
(ZBT  shoitid  be  made  so  that  the  users  get  the  most  powerful  system  for  training,  while  the  Air 
Force  has  the  benefit  of  a  cost-effective  solution. 
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This  study  documents  the  results  of  a  CBT  survey  undertalcen  by  Mei  Associates  for  the 
Human  Resources  Directorate  of  Armstrong  Laboratory  (AL/HR).  The  survey  was  designed  to 
gather  information  from  Air  Force  personnel  about  who  is  doing  CBT,  how  they  are  going  about 
it,  problems  they  are  experiencing,  and  the  consequences  of  those  problems.  This  report  will 
provide  Air  Force  policy  makers,  administrators,  and  researchers,  the  evidence  they  need  to 
begin  improving  the  way  the  Air  Force  currently  develops  and  implements  CBT. 


n.  MEraODS 

One  of  the  most  difficult  tasks  in  gathering  information  about  CBT  problems  is  getting 
people  to  admit  that  problems  exist.  A  unique  feature  of  this  study  is  that  the  researchers 
avoided  asking  participants  about  particular  problems.  Instead,  every  attempt  was  made  to  frnd 
out  about  the  procedures  used  and  the  consequences  of  those  procedures.  This  strategy 
facilitated  obtaining  an  objective  response,  and  provided  the  data  needed  to  make  inferences 
about  current  problems.  If  a  participant  mention^  a  specific  problem,  the  researchers  noted  it; 
however,  researchers  avoided  priming  participants  or  asking  directly  about  potential  problems. 

Sample 

Participants  in  this  study  were  Air  Force  military  and  civilian  training  personnel  directly 
involved  in  CBT  planning,  selection,  or  implementation.  The  sample  included  program 
managers,  contract  managers,  developers,  instructors,  and  programmers. 

Because  CBT  is  a  relatively  new  training  medium,  there  is  no  comprehensive  list  of  Air 
Force  organizations  involved  in  CBT  development  The  population  has  never  been  identifred 
and  statistically  characterized.  For  this  reason,  it  was  impossible  to  select  a  random  sample  for 
this  study.  Instead,  a  noiuandom  sample  was  used  in  which  participants  were  selected  on  the 
basis  of  their  membership  in  the  population  and  their  availability  to  participate. 

Subjects 

There  were  253  participants  in  the  study,  of  which  245  completed  some  portion  of  the 
CBT  survey.*  Of  the  253  participants  the  researchers  conducted  followup  interviews  with  103. 

Participants  bv  Command 

The  participants  represented  14  commands  and  special  operating  agencies,  26  bases,  and 
51  different  organizations  (i.e.,  wings,  groups,  or  offices).  The  major  commands  and  special 
operating  agencies  represented  in  the  study  are  listed  below,  in  decreasing  order  of  the  number 
of  respondents  from  each.  Figure  1  shows  the  number  of  participants  from  each  command. 


*  Of  the  253  personnel  contacted  one  participant  failed  to  turn  in  the  survey 
questionnaire  because  it  was  lost  However,  the  participant  still  provided  data  during  the 
interview  portion  of  the  survey.  Seven  other  participants  completed  the  Personnel  Profile 
Questionnaire  but  did  not  answer  any  survey  questions. 
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o  Air  Training  Command  (ATC) 

o  Strategic  Air  Command  (SAC) 

o  Military  Airlift  Command  (MAC) 

o  Tactic^  Air  Command  (TAC) 

o  Air  Force  Lx>gistics  Command  (AFLC) 

o  Air  Force  Systems  Command  (AFSQ 

o  U.S.  Air  Force  Academy  (USAFA) 

o  Air  Force  Communications  Command  (AFCC) 

o  Air  University  (AU) 

o  Electronic  Security  Command  (ESC) 

o  Air  Force  Space  Command  (AFSPACECOM) 
o  Air  Force  Engineering  and  Services  Center  (AFESC) 
o  Pacific  Air  Forces  (PACAF) 

o  U.S.  Air  Forces  in  Europe  (USAFE) 


Figure  1.  Number  of  Study  Participants  by  (Command 


The  number  of  respondents  per  command  range  from  one  (at  PACIAF  and  USAFE)  to 
123  (at  ATC).  However,  this  graph  should  not  be  interpreted  as  representing  the  actual  number 
of  personnel  who  work  with  CBT  within  each  command;  rather,  it  depicts  those  commands  from 
which  study  participants  were  drawn,  and  the  extent  to  which  the  various  commands  were 
represented.  The  focus  in  selecting  participants  was  on  their  availability  and  experience  using 
some  form  of  CBT  technology. 
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Participants  bv  Rank 

There  were  117  enlisted  participants  (ranging  from  Airman  to  Chief  Master  Sergeant),  70 
officers  (ranging  from  Second  Lieutenant  to  Colonel),  and  66  civilians  (ranging  from  GS-5  to 
GM-14).  Table  1  provides  an  exact  breakdown  of  participants  by  each  rank  or  grade. 

Participants'  CBT  Experience 

The  level  of  CBT  experience  of  the  participants  ranged  from  1  to  150  months.  Most 
participants  had  about  20  months  of  experience  using  CBT.  Survey  participants  were  also  asked 
to  repon  the  amount  of  CBT  experience  they  had  in  particular  job  categories.  These  job 
categories  were: 

o  Manager  of  CBT  program 

o  Developer  of  CBT  courseware 

o  Instructor  in  (DBT  course 

o  CBT  contract  monitor 

o  Staff  computer  systems  expert  involved  in  CBT 


Table  1.  Ranks/Grades  of  Respondents 


1  Enlisted 

Off 

icer 

Civilian  I 

Rank 

Number 

Rank 

Number 

Rank 

Number 

Airman 

1 

2Ll 

2 

GS-5 

1 

_  sk _ 

4 

ILt 

6 

GS-7 

1 

■ESS 

42 

CapL 

28 

GS-8 

1 

TSgt. 

42 

Maior 

21 

GS-9 

12 

MSgt. 

19 

Ll  Col. 

12 

GS-10 

2 

mmmm 

8 

Col. 

1 

GS-11 

16 

1 

GS-12 

22 

GS-13 

2 

GM-13 

6 

GM-14 

3 

TOTAL: 

117 

70 

66 

Note.  Data  from  Personnel  Profrle  (^estionnaiie. 


Table  2  shows  respondents'  average  months  of  experience  at  these  job  positions  by 
command.  In  general,  staff  computer  systems  experts  had  the  most  experience  with  33.5 
months.  Program  managers,  contract  managers  and  developers  each  had  approximately  the  same 
amount  of  experience  (about  2  years).  CBT  Instructors  were  the  least  experienced  participants 
with  19.1  months. 
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Table  2.  Average  Experience  Levels  of  Respondents  by  Command  and  Job  Category^ 


Command 

Manager 

Developer 

Instructor 

Contract 

Monitor 

All 

24.0 

19.1 

24.9 

(163) 

(53) 

(55) 

msM 

ATC 

■KgB 

HPH 

TAC 

28.1 

26.5 

— 

36.3 

... 

(12) 

(12) 

(3) 

SAC 

20.5 

27.4 

26.3 

17.4 

30.2 

(22) 

(18) 

(7) 

(8) 

(6) 

MAC 

24.4 

16.4 

15.0 

29.5 

19.3 

(13) 

(14) 

(2) 

(6) 

(6) 

Others® 

—  No  respondents  in  this  category. 

^  Values  given  =  months  spent  in  the  given  job.  Data  from  Personnel  Profile  Questionnaire. 

^  Values  in  parentheses  are  the  total  number  of  respondents  to  the  question  regarding  months 
experience  for  the  given  job  categoiy.  Many  respon^nts  were  holding  or  had  held  more  than 
one  type  of  position. 

c  "Others"  category  includes  USAFA,  AFSPACECOM,  AFCC,  AFLC,  AFSC,  ESC,  USAFE, 
AU,  and  AFESC. 


Survey  Development 

The  survey  was  designed  and  structured  so  that  groups  of  questions  were  answered  only 
by  personnel  who  had  the  required  background  knowledge.  Branching  questions  were  used 
extensively  to  route  respondents  around  sections  of  the  questionnaire  and  specific  questions  they 
were  not  qualified  to  answer.  For  example,  at  the  beginning  of  each  major  section  of  the  survey 
-  Planning,  Selection,  Development,  Validation,  Implementation,  and  Maintenance  •- 
participants  were  asked  if  they  had  been  involved  in  that  activity;  if  they  had  no  experience,  they 
were  branched  to  the  next  section.  This  feature  enabled  the  long  questionnaire  to  be  completed 
within  a  reasonable  time  (approximately  one  hour)  by  most  participants.  Another  feature  of  the 
survey  which  helped  shorten  the  response  time  was  that  questions  ^ut  various  CBT  procedures 
such  as  listing  of  system  components  or  giving  reasons  or  causes  for  stmiething  happening  were 
presented  in  checklist  form.  This  design  allowed  respondents  to  simply  check  off  the  responses 
that  described  their  situation  and  enabled  the  researchers  to  easily  collect  detailed  data.  Figure  2 
depicts  a  set  of  typical  branching  questions;  a  complete  copy  of  the  survey  instrument  can  be 
found  in  the  Appendix. 
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Figure  2.  Sample  Survey  Questions 


9.  Was  a  formal  job/task  analysis  performed  as  part  of  the  CBT  planning 
process? 

Yes  No 

10.  If  9  =  No,  why  not?  (Check  the  appropriate  lesponse/s.) 

_  There  wasn't  enough  time. 

_  There  were  not  enough  funds. 

_  Personnel  were  not  available  to  conduct  the  analysis. 

_  Personnel  (subject  matter  experts,  etc.)  were  not  available  to  provide 

the  data  necessary  to  complete  the  analysis. 

_  Jobs  and  tasks  were  derived  from  current  course  material  and/or 

manuals. 

_  The  job/task  analysis  completed  previously  was  current 

_  Other: _ 


The  development  of  such  an  extensive  survey  required  information  from  various  sources. 
The  research  team  reviewed  a  broad  spectrum  of  literanirc  on  CBT  planning,  selection  of  CBT 
systems,  C3T  development,  implementation  and  validation  procedures,  and  the  maintenance  of 
CBT  courseware  and  systems  (e.g.,  Gery,  1987;  Alessi  and  Trollip,  1985;  Kearsley,  1987; 
Schlecter,  Burnside  and  Thomas,  1987).  In  addition  to  the  literature  review,  the  members  of  the 
research  team  used  their  own  experiences  in  CBT  as  a  starting  point  for  development  of  the 
questionnaire.  Armstrong  Laboratory  scientists  commented  on  initial  draft  versions  of  the 
questionnaire  and  arranged  for  a  tryout  of  the.  instrument  and  data  collection  procedures  with  a 
nearby  command  using  CBT.  lliis  earlier  version  of  the  questionnaire  consisted  of  198 
questions  divided  into  five  sections:  Planning,  Selection,  Development,  Validation,  and 
^plementation.  It  was  admiiustered  to  47  subjects.  The  second  version  was  restructured 
slightly  and  included  additional  questions  to  cover  potential  problem  areas  not  fully  addressed  in 
the  initial  version  (such  as  courseware  maintenance).  Otherwise,  questions  on  the  two  versions 
were  similar.  The  revised  version  consisted  of  226  questions  divided  over  six  sections 
(including  a  new  section  on  Maintenance),  and  was  administered  to  206  subjects.  Both  versions 
also  contained  a  Persoimel  Profile  section  for  demogrqrhic  data  and  a  Discussion  section,  to 
cover  respondents'  areas  of  concern  or  interest  which  had  not  been  addressed  by  the  questions. 
For  analysis,  data  from  the  two  versions  of  the  questionnaire  were  ctmibined. 

Branching 

Sample  size  varied  among  many  survey  questions  as  reflected  by  differences  between 
values  of  n  in  Figures,  Tables  and  the  Appendix.  This  is  due  to  the  questionnaire's  branching 
structure  that  channeled  respondents  to  appropriate  questions  based  on  individuals'  qualifications 
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and  experience.  In  addition,  a  few  respondents  simply  skipped  some  questions.  The  percentages 
represent  the  number  of  respondents  who  chose  a  question  option  compared  to  the  number  of 
individuals  eligible  to  respond  (i.e.,  adjusted  for  branching  and  skipped  answers). 

Figure  2,  for  example,  shows  a  typical  survey  branching  question.  These  data  are  from 
survey  questions  9  and  10.  A  total  of  47  respondents  answering  No  to  question  9  were  eligible 
to  answer  question  10.  However,  the  option  The  jobftask  analysis  completed  previously  was 
current,  did  not  appear  on  the  earlier  version  of  the  questionnaire,  therefore  fewer  respondents 
(38)  were  eligible  to  select  it  The  actual  distribution  of  percentages  for  these  questions  can  be 
seen  in  Figure  4  in  the  Results  section  of  this  report 

Survey  Procedures 


Data  Gathering 

After  the  survey  design  was  completed  and  approved  by  the  Armstrong  Laboratory,  it 
was  necessary  to  iden^y  participants;  this  was  primarily  the  responsibility  of  the  Laboratory 
scientists.  A  comprehensive  list  of  Air  Force  individuals  and  groups  involved  with  CBT  was  not 
available.  Therefore,  the  Laboratory  scientists  with  the  cooperation  of  major  command  points 
of  contact  (POC)  identified  and  initially  contacted  potential  study  participants  using  snowball 
sampling  techniques.  Organizations  developing  (TBT  in-house  were  of  particular  interest 
Potential  participants  were  contacted  and  survey  participation  set  up  as  the  Laboratory  learned 
about  them.  (Dnce  an  organization  fitting  the  profile  was  identified.  Laboratory  scientists 
designated  a  POC  at  the  site  and  determined  the  number  of  individuals  who  would  participate  in 
the  study.  The  researchers  sent  surveys  to  the  POC  at  the  site,  who  was  in  charge  of  their 
distribution  and  return.  This  approach  resulted  in  a  96.8%  response  rate  for  the  survey. 

During  early  phases  of  data  collection  the  researchers  visited  each  base  that  participated 
in  the  survey.  As  trends  in  responses  to  interview  questions  were  established,  i.e.,  respondents 
were  discussing  the  same  matters  over  and  over,  visits  were  restricted  to  those  organizations 
which  were  different  from  the  ones  previously  visited  in  terms  of  type  of  CBT  system,  major 
command,  or  other  characteristics.  Questionnaires  were  mailed  back  to  the  researchers,  who 
reviewed  them  and  identified  participants  for  follow-up  interviews  to  clarify  responses. 
Sometimes  follow-up  interviews  were  scheduled  because  of  the  extensive  CBT  experience  of 
some  respondents.  The  feeling  was  that  the  more  experience  a  respondent  had,  the  better  he  or 
she  would  be  able  to  describe  what  was  going  on  in  Air  Force  CBT.  In  some  cases.  Laboratory 
scientists  accompanied  the  research  team  on  site  visits.  Armstrong  Laboratory  personnel  did  not 
participate  in  the  interviews.  The  researchers  sought  to  eliminate  any  potential  influence  by 
another  Air  Force  individual  during  the  survey  so  that  the  responses  (especially  during  the 
interview)  were  those  of  the  respondent  alone,  not  necessarily  official  policy. 

Setting  Up  the  Database 

SPSS/PC  (version  4.0)  was  used  for  statistical  analysis.  The  coding  strategy  assigned  a 
variable  for  each  possible  response  to  each  question.  Each  variable  was  coded  as  "0"  for  No,  "1" 
for  Yes,  "8"  if  a  question  was  skipped  because  the  respondent  had  been  branched  around  it,  and 
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"9"  if  a  question  was  skipped  because  the  respondent  chose  not  to  answer  it  or  misinterpreted 
directions. 

Several  survey  questions  were  designed  to  have  multiple  responses.  For  these  questions, 
each  response  was  designated  as  a  unique  variable.  Thus,  data  from  the  questionnaire,  which 
contained  226  questions,  were  coded  as  928  separate  variables.  The  data  matrix  (n  x  the  number 
of  variables)  was  245  x  928. 

Data  PrepaiatiQn 

Some  respondents  did  not  answer  the  survey  questions  as  directed.  For  example,  for 
questions  where  a  Yes  or  No  response  was  called  for,  some  respondents  circled  both.  In  other 
cases,  respondents  would  simply  skip  questions,  either  because  they  misinterpreted  instructions 
or  because  they  did  not  wish  to  respond.  Before  the  survey  data  were  entered  into  the  database, 
these  unexpected  responses  had  to  be  reconciled  so  that  each  variable  for  every  respondent  had 
only  a  single  recorded  response. 

Rules  were  devised  to  systematize  the  reconciliation  process.  For  example,  in  cases 
where  a  participant  circled  both  Yes  and  No^  the  response  which  was  more  indicative  of  a 
problem  was  coded.  Or,  in  cases  where  circling  Yes  or  No  allowed  the  respondent  to  branch 
around  a  set  of  questions,  but  follow<up  questions  were  answered  anyway,  the  response  to  the 
original  question  was  changed  to  be  consistent  with  their  answering  the  foUow'Up  questions. 
Reconciling  the  data  involved  checking  each  survey  individually  for  accuracy,  coding  skipped 
questions  where  required,  and  verifying  inconsistent  responses. 

During  the  reconciliation  process,  steps  were  taken  to  insure  compatibility  between 
responses  from  the  initial  and  final  versions  of  the  survey.  In  cases  where  questions  existed  on 
one  version  but  not  on  the  other,  the  corresponding  variables  were  coded  as  ”8"  -  skipped  for  all 
cases  where  the  question  was  not  included!  This  allowed  the  researchers  to  work  with  a  single 
consolidated  database  of  information  from  botii  versions. 

Data  Analysis 

Data  analysis  focused  on  providing  descriptive  data  and  comparing  relationships  at  the 
aggregate  respondent  level.  For  issues  of  special  interest,  data  were  collapsed  and  assessed  for 
specific  demographic  groups  such  as  major  commands  and  various  experience  levels. 

Analyses  were  conducted  in  two  stages  due  to  the  large  amount  of  data  collected  and  the 
size  of  the  data  matrix. 

Stage  1.  Descriptive  statistics  were  run  during  the  initial  analysis.  These  included 
computing  frequencies,  means,  medians,  and  modes,  depending  on  the  level  of  measinement. 
These  statistics  provided  a  description  of  the  data,  an  assessment  of  relationships  between  key 
variables,  and  a  reduced  data  set  Results  from  this  analysis  were  used  for  three  primary 
purposes:  1)  to  provide  a  preliminary  description  of  the  data;  2)  to  identify  issues  for  further 
analysis;  an^  3)  to  eliminate  variables  with  too  few  respondents  to  provide  useful  information. 
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Stage  2.  The  second  stage  of  analysis  involved  creating  cross-classification  tables,  both 
simple  and  elaborated,  using  the  reduced  variable  set  from  Stage  1  to  provide  a  more  detailed 
analysis  of  relationships  between  variables. 

Handling  of  Other  Responses 

The  checklist  structure  of  this  survey  ensured  that  much  of  the  infcmnation  queried  was 
already  in  the  text  of  the  questions.  Participants  could  simply  check  applicable  responses. 
However,  in  case  participants  wished  to  add  their  own  responses,  all  checklist  questions 
contained  an  Other  option  where  respondents  could  record  any  relevant  information  which  was 
not  already  listed.  During  the  data  entry  phase  of  the  study.  Other  responses  were  recorded  in 
the  database.  After  firequencies  were  computed.  Other  responses  to  a  given  question  were 
examined  if  more  than  30%  of  respondents  had  indicated  one  for  that  question.  If  several 
subjects  mentioned  the  same  type  of  information,  the  trend  was  investigated. 

Qualitative  Data 

Numerous  qualitative  data  were  gathered  from  the  Discussion  section  of  the  survey  and 
from  the  interviews.  To  reduce  these  data  to  a  mrae  manageable  form,  information  was 
categorized  into  problem  areas.  A  separate  coding  scheme  was  developed  for  each  and  tested  for 
reliability  using  phi.  The  inter-rater  reliability  was  .70  for  the  Discussion  section,  and  .46  for  the 
interviews. 


ffl.  RESULTS 

The  researchers'  goal  was  to  conduct  a  survey  of  Air  Force  organizations  involved  in 
developing  and  implementing  CBT  in  order  to  reveal  any  problems  that  should  be  corrected,  and 
successful  practices  which  could  be  modeled  by  other  organizations.  This  chapter  describes  nine 
CBT  issues.  In  the  following  sections,  each  issue  will  be  identified  and  explained: 

o  CBT  plaiuiing  procedures 

o  Management  of  CBT 

o  Preparation  of  CBT  development  persormel 

o  Effective  use  of  CBT 

o  Efficiency  of  the  CBT  development  process 

o  Standardization 

o  Validation 

o  Implementation 

o  Maintenance 


CBT  Plarming  Procedures 

Plaiuiing  is  a  critical  function  in  all  CBT  development  efforts  (Andrews  and  Trainor, 
1987).  It  cannot  be  overemphasized  that  proper  planning  can  and  will  eliminate  many  problems 
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encountered  later  in  a  CBT  program.  The  elements  of  planning  for  CBT  are  similar  to  planning 
for  any  other  successful  venture  (Outer,  1991;  Kemner-Richardson,  Lamos  and  West,  1984). 
First,  the  organization  must  make  sure  that  CBT  is  the  appropriate  instructional  medium  for  their 
training  requirements  (i.e.,  they  need  to  conduct  some  kind  of  media  analysis).  Second,  the 
required  resources  (people,  equipment,  money,  facilities,  students,  etc.)  must  be  budgeted  for 
realistically.  Finally,  the  managers  of  the  CBT  effort  must  make  sure  that  the  people  who  will 
actually  be  developing  CBT  courseware,  and  implementing  the  CBT  system  are  properly 
equipped  to  do  their  jobs.  That  is,  courseware  developers  need  to  have  up-to-date  and  complete 
course  development  materials  available  (such  as  relevant  data  on  job/task  analyses,  trainee 
characteristics,  training  needs  assessment,  and  training  objectives).  In  addition,  facilities  need  to 
be  prepared  to  house  the  planned  computer  hardware  in  the  desired  configuration,  and  the  staff 
who  be  operating  the  CBT  system  must  be  prepared  for  its  integration  into  the  training 
environment 

Although  planning  is  important  to  developing  CBT  on  time  and  implementing  it 
effectively,  our  findings  show  that  the  Air  Force  lacks  specific  policy  or  guidelines  for  CBT 
planning.  Currently,  CBT  planners  must  rely  on  the  policy  which  they  can  extract  from  the 
Instructional  Systems  Development  (ISD)  process.  Although  the  ISD  process  can  be 
successfully  adapted  to  any  instructional  medium,  there  are  peculiarities  unique  to  CBT  which 
make  planning  for  it  different  from  tradititmal  training  plaiming.  These  peculiarities  include  the 
need  to  purchase,  become  familiar  with,  and  install  large  quantities  of  new  and  expensive 
equipment,  the  preparation  of  faculty  and  staff  for  their  roles  in  a  C3T  classroom,  and 
predetermining  specific  details  of  CBT  lesson  content  to  facilitate  courseware  development 

Several  of  the  major  planning  steps  of  the  ISD  process  were  investigated.  Table  3  shows 
the  results  for  those  respondents  whose  oiganizations  performed  each  planning  step.  Few 
respondents  (27.4%)  reported  their  organization  complet^  all  six  steps.  However,  almost  half 
(48.9%)  reported  that  their  organization  completed  at  least  five  steps  with  the  most  commonly 
omitted  step  being  Media  Analysis,  a  critical  factor  in  CBT  planning.  This  survey  focused  on 
assessing  orgaiuzations'  performance  on  seven  important  issues  in  CBT  planning: 

o  Training  Needs  Assessment 

o  Job/Task  Analysis 

o  Analysis  of  Trainee  Characteristics 

o  Assessment  of  Facility  Requirements 

o  Media  Analysis 

o  Development  of  Training  Objectives 

o  Budgeting 

This  report  will  show  why  each  of  these  issues  is  important  for  (TBT  planning,  and  how  these 
issues  are  being  addressed  by  Air  Force  organizations. 
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Table  3.  Performance  of  CBT  Planning  Steps  by  Command^ 


Command 

Training 

Naeda 

Aawfsmcnt 

JoIVTaric 

Analytia 

Analyaiaor 

Trainea 

Charactariatka 

Facflttka 

RaqnJraniaBla 

Madia 

Analydi 

Training 

Objactivaa 

IWWM 

77.6 

67.1 

83.1 

64.1 

50.0 

86.9 

78.0 

67.8 

81.7 

76.3 

56.1 

85.0 

cSlHi 

90.9 

81.8 

55.6 

60.0 

70.0 

90.9 

85.7 

71.4 

92.9 

57.1 

28.6 

85.7 

75.0 

56.3 

87.5 

56.3 

31.3 

87.5 

IRfRBH 

iBBEhI 

72.1 

66.7 

86.0 

53.5 

51.2 

90.7 

^  Percentage  of  respondents  who  performed  the  given  step.  Data  from  survey  questions 
4,9,14,18, 22, 25  and  Persoimel  Profile  Questionnaire. 

Numbers  in  parentheses  *  number  of  respondents  from  the  command, 
c  "Others"  category  includes  USAFA,  AFSPACECX)M,  AFCX^,  AFLC,  AFSC,  ESC,  USAFE, 
AU,  and  AFESC. 


Training  Needs  Assessment 

Training  Needs  Assessment  is  normally  performed  when  an  organization  suspects  some 
discrepancy  between  what  is  being  trained  and  what  should  be  trained  (Kaufman  and 
Thiagarajan,  1987).  A  large  number  of  respondents  (77.6%)  indicated  that  their  organizations 
performed  Training  Needs  Assessment  This  number  is  surprising  in  light  of  the  reasons  given 
for  not  performing  such  an  assessment  Figure  3  lists  reasons  why  a  Training  Needs  Assessment 
was  not  conducted.  The  majority  of  respondents  (59.4%)  indicate  that  they  used  current  data  or 
that  a  previous  Training  Needs  Assessment  was  current  (8.7%).  This  implies  that  CBT  was  pan 
of  a  media  upgrade  for  the  training  system,  but  the  low  number  reporting  Media  Analysis  (50%) 
is  contradictory.  Other  respondents  indicated  that  personnel  woe  not  available  to  conduct  the 
study,  no  funds  were  available,  and/or  there  was  not  enough  time  to  conduct  the  study. 
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Figure  3.  Reasons  Why  Training  Needs  Assessment  Not  Performed^ 
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CUnant 


Note.  Expressed  as  percentages  of  respondents  citing  the  reason  given  for  not 
performing  a  Training  Needs  Assessment.  Data  from  survey  questions  4  &  5. 


Job/Task  Analysis 

Instructional  designers  agree  that  the  foundation  for  quality  instruction  is  laid  by  a  good 
Job/Task  Analysis  (Merrill,  1987).  Job/Task  Analysis  normally  involves  listing  all  job  tasks, 
equipment  or  materials  involved  with  each  task,  the  conditions  under  which  tasks  are  performed, 
and  the  standards  of  performance  which  must  be  met  This  material  constimtes  a  formal 
description  of  what  the  trainee  will  ultimately  be  required  to  do  on*the-job.  The  principles  of 
ISD  expressed  in  AFM  50-2  state,  'To  know  precisely  what  instruction  is  needed,  you  must  first 
know  what  tasks  and  knowledges  make  up  the  job."  However,  Table  3  shows  Aat  32.9%  of 
respondents  did  not  perform  a  Job/Task  Analysis.  The  respondents  cited  various  reasons  for  not 
conducting  such  an  analysis  (Figure  4).  The  most  commonly  cited  reason  was  that  current 
course  materials  were  us^  (55.3%);  similar  reasons  were  given  for  not  performing  a  Training 
Needs  Assessment  (59.4%).  This  approach  might  be  acceptable  if  the  training  requirements  have 
not  changed.  However,  only  39.5%  of  those  who  did  not  perform  a  Job/Task  Analysis  said  that  it 
was  because  previous  materials  were  current  If  such  a  high  percentage  of  respondents  reported 
that  materials  were  not  current,  and  no  Job/Task  Analysis  was  performed,  one  might  speculate  as 


2  Whenever  there  are  differences  in  n  due  to  branching  or  an  eariier  version  of  the 
questionnaire,  the  number  of  respondents  who  answered  the  item  versus  the  total  number  of 
respondents  eligible  to  answer  the  question  is  listed  next  to  the  bar.  For  a  full  explanation  see 
Branching  in  the  Methodology  section  of  the  report 
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to  whether  or  not  the  CBT  being  developed  matches  the  job  requirements.  Otha  reasons  given 
for  not  performing  a  formal  Job/Task  Analysis  include  not  having  enough  personnel  to  collect  or 
analyze  the  data.  This  theme  reappears  throughout  other  findings  in  this  reptnt.  Lack  of 
personnel  or  not  being  able  to  maintain  a  CBT  team  appears  to  be  ctnnmon  in  Air  Force  CBT 
projects  and  can  adversely  affect  the  CBT  effOTt  Air  Force  CBT  personnel  may  not  be  aware 
that  Job/Task  Analysis  is  a  critical  component  of  any  training  development  effort 


Figure  4.  Reasons  Why  Formal  Job/Task  Analysis  Not  Performed 
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Note.  Expressed  as  percentages  of  respondents  who  did  not  perform  a 
Job/Task  Analysis.  Data  from  survey  questions  9  &  10. 


Analysis  of  Trainee  Characteristics 

A  large  number  of  respondents  (83.1%)  indicated  that  their  organizations  analyzed 
trainee  characteristics.’  This  analysis  usually  involves  determining  the  skills,  knowledge, 
abilities,  and  attitudes  of  prospective  students  and  is  used  with  Jt^^ask  Analysis  data  to 
determine  what  needs  to  be  taught,  i.e.,  the  difference  between  what  is  required  and  what  is 
already  known  (Dick  and  Carey,  1985).  It  can  also  be  used  to  determine  if  using  certain 


’  Data  from  survey  question  14. 
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instructional  strategies  or  media  would  be  effective.  As  part  of  CBT  planning,  it  is  a  small,  but 
significant  contribution  to  the  decision-making  process. 

Facility  Requirements  Assessment 

Survey  data  indicate  training  personnel  do  not  always  assess  facility  requiier- 'nts  for 
CBT.^  They  frequently  assume  that  existing  facilities  will  be  sufficient,  the  computer  equipment 
may  already  be  in  place  or  else  can  be  installed  anywhere  (the  latter  is  often  true  when 
microcomputers  are  used).  These  assumptions  can  cause  problems  later  when  it  turns  out  that 
changes  are  required,  i.e.,  where  facilities  are  insufficient,  existing  equipment  caimot  be  used,  or 
existing  equipment  is  incompatible  with  the  courseware.  In  general,  it  is  far  more  efficient  to 
identify  facility  requirements  at  the  beginning  of  a  project  and  to  plan  accordingly,  than  to  be 
confronted  with  delays  later. 

Various  reasons  were  cited  for  why  a  Facilities  Requirements  Assessment  was  not 
performed.  Some  respondents  indicated  that  because  they  were  already  using  microcomputers 
such  an  assessment  was  unnecessary  (15.4%).  Others  indicated  that  it  was  generally  unnecessary 
(13.5%)  or  that  personnel  were  not  available  to  conduct  the  planning  (13.5%)  or  provide  the  data 
(11.5%)  or  else  money  (9.6%)  or  time  (9.6%)  was  not  available.  Over  44%  of  respondents 
stated  the  organization  could  not  condua  the  assessment  This  is  interesting  because 
respondents  recognized  the  need  to  perform  such  an  assessment  although  tlwir  organizations 
were  unable  to  complete  the  CBT  planning  step. 

Mgdi^  Analysis 

As  part  of  CBT  planning.  Media  Analym  is  the  least  frequently  performed  step  (50%). 
Media  Analysis  usually  involves  determining  what  types  of  instructional  media  would  be  most 
appropriate  for  delivery  of  training  (AFP  50-58,  Vol.  IV;  Gagne  and  Briggs,  1979;  Dick  and 
(jarey,  1985).  Often  mote  than  one  medium  can  be  used.  Ideally,  the  decision  should  be  based 
on  which  medium  or  media  mix  best  meets  training  requirements.  Other  factors  such  as 
cost-effectiveness  may  also  affect  the  decisitMi.  CBT  may  be  selected  because  of  its  ability  to 
provide  interactive,  self-paced  instruction  to  students,  to  present  st^histicated  graphics  or 
animation,  or  any  of  its  other  special  media  characteristics.  Reasons  not  to  select  CBT  may 
include  its  high  irutial  cost,  frequently  changing  instructional  objectives,  infrequent  delivery,  or 
the  small  size  of  the  group  to  be  trained.  Some  kind  of  media  analysis  would  prove  beneficial 
because  of  the  large  investment  of  personnel,  time  and  money  required  to  produce  (TBT. 


<  64. 1%  reported  that  facilities  requirements  are  assessed.  See  Table  3. 
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Figure  5.  Reasons  Why  Media  Analysis  Not  Performed 


n  =  70 


Note.  Expressed  as  percentages  of  respondents  who  did  not  perform  a  media  analysis. 
Data  from  survey  questions  22  &  23. 


Of  the  respondents  who  indicated  that  Media  Analysis  was  not  performed.  Figure  6 
shows  the  ovCTwhelming  reason  (77.1%)  was  that  media  were  predetermined  by  a  higher 
authority.  This  raises  the  questions:  Who  predetermined  it?  What  was  the  decision  based  on? 
Was  a  better,  or  cheaper  alternative  considered?  Were  any  alternatives  considered?  As  stated 
earlier,  the  Air  Force  has  few  guidelines  for  CBT  planning  and  selection.  It  may  be  that  there 
were  reasons  to  select  CBT  that  were  not  apparent  to  resptmdents.  During  the  interviews 
conducted  in  connection  with  the  survey,  respondents’  comments  included:  "Often  CBT  is  used 
when  some  other  medium  would  be  more  appropriate.".  .  .  "The  Air  Force  larks  a  tool  to 
determine  when  CBT  is  the  proper  medium.". .  .  "The  CAI  Handbook  is  not  adequate  for  this 
purpose.". . .  "CBT  was  selected  because  higher  authorities  wanted  it"  Overall,  the  data  suggest 
either  an  incomplete  understanding  of  CBT  planning,  or  respondents  were  not  aware  of  the 
context  of  prior  decision-making. 
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Obiecrives  Development 


Nearly  everyone  responding  to  the  survey  (86.9%)  reported  that  their  organization 
developed  objectives  on  which  CBT  was  based.  One  would  expect  this  to  be  the  case,  since  the 
use  of  criterion-referenced  objectives  seems  to  be  the  one  point  on  which  there  is  little 
disagreement  throughout  most  military  training  organizations.  In  those  few  instances  where 
objectives  were  not  developed,  the  overwhelming  majority  reported  that  it  was  because  they 
made  use  of  existing  objectives  (73.7%).  Only  one  respondent  reported  that  objectives  were  not 
required. 

Budgeting 

One  result  of  the  CBT  planning  phase  is  a  program  budget  Table  4  shows  which  Air 
Force  personnel  were  involved  in  developing  the  CBT  budget  and  their  roles  according  to  their 
report^  job  categories.  Note  that  relatively  few  respondents  from  any  job  category  provided 
data  about  facilities  requirements  or  conducted  cost-benefit  analyses,  two  important  CBT 
planning  criteria.  Based  on  the  infrequent  performance  of  facility  requirements  assessment 
(64.1%)  and  media  analysis  (50.0%),^  it  appears  that  these  processes  were  not  completely 
considered  when  formulating  the  CBT  budget  Both  of  these  plarming  steps  can  affect  the 
budget.  Problems  associated  with  CBT  installation  such  as  rewiring  a  classroom,  or  moving 
lighting  fixtures  or  electrical  outlets  may  appear  later.  Such  problems  could  be  avoided  by 
assessing  the  facilities  requirements  during  CBT  planning. 

Cost-benefit  analysis  is  the  commonly  accepted  procedure  for  assessing  alternative 
approaches  to  a  problem.  Cost-benefit  analyses  fin^uently  raise  questions  early  in  a  CBT 
program,  and  can  be  show  stoppers  if  delayed  until  later  stages  of  a  program.  Table  4  suggests 
that  neither  of  these  planning  steps  was  given  much  attention  by  most  of  the  respondents  and  that 
many  of  those  involved  in  CBT  planning  may  not  be  experienced,  or  may  lack  the  proper 
guidance  to  follow  in  performing  such  analyses. 


^  See  Table  3  for  the  percentages  of  respondents  performing  each  planning  step. 
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Table  4.  Percentages  of  Job  Categories  Performing  Various  Budgeting  Roles* 


Budgeting 

Role 

Program 

Manager 

(n=61)'» 

Contract 

Manager 

(n=18) 

Instructional 

Developer 

{n=51) 

Instructor 

(ns20) 

SME 

(n=33) 

Hardware 

Acquisition 

Input 

83.6 

77.8 

82.4 

90.0 

87.9 

1  Software 

1  Acquisition 

1  Input 

93.4 

83.3 

92.2 

90.0 

90.9 

1  Facilities 

1  Input 

32.8 

50.0 

25.5 

35.0 

33.3 

1  Personnel 

I  Requirements 

1  Input 

70.5 

72.2 

72.5 

70.0 

75.8 

1  Project 
Completion- 
1  Time  Input 

83.6 

94.4 

86.3 

95.0 

90.9 

1  Team 

Training 

Requirements 

65.6 

61.1 

68.6 

65.0 

63.6 

Cost-Benefit 

Analyses 

49.2 

50.0 

51.0 

60.0 

45.5 

1  Development 

1  of  the 

1  Budget 

36.1 

55.6 

35.3 

45.0 

36.4 

^  Percentages  of  respondents  with  the  given  job  title  who  perftvmed  a  given  role.  It  was 
possible  for  a  respondent  to  fill  multiple  job  positions  and  to  p^orm  multiple  budgeting  roles. 
Data  from  survey  questions  2, 28  &29. 

^  Numbers  in  parentheses  =  number  in  the  job  category  who  reptmed  involvement  in 
budgeting. 


One  surprising  result  shown  in  Table  4  is  the  relatively  high  degree  of  involvement  of 
instructors  and  SMEs  in  providing  information  about  hardware  and  software.  The  researchers 
feel  that  the  respondents  may  have  meant  that  they  gave  advice  regarding  the  type  of  hardware 
and  software  rather  than  actually  developing  the  CBT  budget  for  these  items.  Usually,  those 
most  familiar  with  the  cost  of  computer  systems  would  be  expected  to  provide  input  regarding 
the  costs  of  hardware  and  software.  In  Air  Force  CBT  efforts  anyone  could  be  the  source  of  this 
information  including  instructors  and  SMEs.  However,  the  budget  for  a  CBT  program  would 
best  be  developed  by  the  program  manager  or  contract  manager. 
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In  spite  of  the  involvement  of  a  wide  variety  of  personnel  in  the  CBT  budgeting  process, 
49.4%  of  respondents  said  that  the  budget  did  not  accurately  anticipate  costs.^  Several  problem 
areas  were  identified  and  clustered  into  three  major  categories:  1)  cost  estimates  associated  with 
hardware  and  software  acquisition;  2)  time  allocated  for  various  development  functions;  and  3) 
personnel  turnover. 


Figure  6.  Sources  of  Unexpected  Cost 


ns38 
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Note.  Expressed  as  percentages  of  respondents  who  indicated  that  the 
budget  did  not  accurately  anticipate  costs.  Data  from  survey 
questions  30  &  31. 


Figure  6  displays  a  number  of  items  related  to  unexpected  costs.  In  the  category  of  cost 
estimation  six  items^  related  to  the  acquisition  of  CBT  hardware  and  software  were  associated 
with  cost  overruns.  Initially,  it  seems  that  titose  organizations  that  underestimated  the  costs  of 
hardware  (IS.8%)  and  software  (15.8%)  could  have  done  a  better  job  of  predicting  computer 
systems  costs.  However,  when  viewed  tt^ther  with  the  other  related  items,  namely  "bought 
more  hardware"  (34,2%),  "bought  more  software"  (28.9%),  "underestimated  hardware 


®  Data  from  survey  question  30. 

These  are  the  frrst  six  items  displayed  in  Figure  6. 
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maintenance  costs"  (13.2%)  and  "underestimated  software  maintenance  costs"  (18.4%),  there 
appears  to  be  no  clear  guidance  for  estimating  costs. 

There  are  also  problems  associated  with  developing  CBT  within  the  original  time 
estimate.  When  asked  whether  the  CST  budget  accurately  anticipated  costs,  respondents 
indicated  that  there  were  four  sources  of  unexpected  costs  related  to  completing  a  project  on 
time.  Over  half  (SS.3%)  of  the  respondents  indicated  that  CBT  development  took  longer  than 
expected.  CBT  review  time  was  exceeded  by  28.9%  of  the  respondents,  and  revisions  caused 
unexpected  production  delays  (31.6%).  These  problems  relate  to  the  relative  inexperience  of  the 
CBT  development  team  and  ineffective  project  planning  (Jay,  Bernstein  and  Gunderson,  1987). 
Peihaps  the  most  significant  item  was  that  36.8%  of  the  respondents  reported  that  the  training 
requirements  changed,  causing  courseware  revisions.  If  this  information  is  considered  with  the 
fact  that  only  67.1%  of  respondents  reported  completing  a  Job/Task  Analysis  (see  Table  3),  the 
effect  of  not  properly  defining  the  training  requirements  beftne  starting  CBT  can  be  better 
appreciated. 


Figure  7.  Effects  of  Funding  Shortfalls 
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Note.  Expressed  as  percentages  of  respondents  who  indicated  that 
funding  shortfalls  caused  the  given  effect.  Data  fnmi  survey 
questions  32  &  33. 
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The  third  problem  related  to  CBT  budgeting  was  personnel  turnover.  Respondents 
reported  that  the  cost  of  training  CBT  development  team  personnel  (15.8%)  was  not  factored 
into  the  original  budget  In  addition,  42.1%  reported  that  additional  training  was  required 
because  of  the  turnover  of  team  personnel.  While  the  cost  of  traiiting  team  pnsonnel  might  not 
be  a  major  expense  in  a  CBT  budget  it  reflects  persoimel  turnover  that  is  associated  with  many 
Air  Force  CBT  projects.  Personnel  turnover  can  seriously  affect  many  aspects  of  a  CBT  project 
such  as  schedu^g  or  causing  additional  revisions  because  of  failure  to  adhere  to  established 
patterns  and  practices. 

Poor  estimation  of  CBT  costs  or  simply  the  lack  of  availability  of  funds  can  have  a 
serious  impact  on  (TBT  projects.  The  survey  investigated  the  results  that  funding  shortfalls  had 
on  projects  (Figure  7)  and  found  three  major  consequences:  1)  projects  are  put  on  hold  (41.4%), 
2)  less  equipment  is  purchased  than  originally  estimated  (43.6%),  or  3)  the  scope  of  the  effon  is 
reduced  (38.5%).  Each  alternative  is  an  acceptable  approach  to  accommodating  funding 
shortfalls.  However,  there  were  three  items  reported  which  might  negatively  impact  the  quality 
of  Air  Force  CBT  lessons.  Namely,  these  are  the  tendency  to  reduce  the  lengA  of  individud 
lessons  (7.7%),  a  reduction  of  interactivity  within  the  lessons  (23.1%),  and  a  reduction  in  the 
number  of  graphics  used  (25.6%).  None  of  these  alternatives  is  acceptable  if  it  downgrades  the 
training. 

Summary  of  CBT  Planning  Procedures 

In  summary.  Air  Force  CBT  planning  appears  to  be  inconsistent  Planning  steps  are 
often  omitted  or  are  performed  in  a  cursory  fashion.  The  reasons  given  for  why  steps  are  not 
performed  suggest  that  the  basic  problem  is  a  lack  of  understanding  of  what  CBT  planning 
requires,  rather  than  lack  of  resources  for  planning.  The  net  effect  of  not  completing  CBT 
planning  steps  is  that  all  subsequent  processes  are  affected.  Clear  policy  and  usable  CBT 
planning  guidelines  are  needed.  Air  Force  training  personnel  may  not  have  the  training  and 
experience  to  know  how  to  plan  without  additional  assistance. 

Management  of  CBT 

Managing  CBT  development  is  difierent  from  managing  development  of  other  training 
media  (MacNiven,  1987;  Air  Training  Command,  1988).  To  begin  with,  CBT  is  a  still  emerging 
technology.  In  many  cases,  prospective  CBT  manage  are  not  sure  what  is  required  for  CBT 
development,  or  how  to  support  the  technology.  Second,  CBT  develq)ment  requires  not  only 
knowledge  about  the  specific  subject  matter  and  training  methods,  but  also  familiarity  with 
computer  technology.  CBT  managers  need  to  be  willing  to  learn  about  and  prepare  themselves 
for  this  new  technology. 

The  survey  did  not  ask  direct  questions  about  management  problems  on  the  assumption 
that  participants  should  not  be  placed  in  a  position  of  having  to  respond  to  potentially  sensitive 
issues.  Surprisingly,  however,  respondents  were  free  with  their  ctnnments  about  management 
when  given  the  opportunity  to  express  themselves:  31.2%  commented  about  management  in  the 
discussion  section  of  the  survey,  and  66%  did  so  during  the  follow-up  interviews.  These 
comments  and  indirect  indications  from  the  survey  suggest  that  Air  Force  CBT  managers  are  not 
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efficiently  managing  CBT  projects,  including,  for  example: 

o  providing  clear  guidance, 

o  coordinating  development  team  efforts, 

o  meeting  deadlines,  and 

o  being  familiar  with  CBT  technology. 

Guidance  in  CBT  Development 

One  indicator  that  managers  are  not  providing  sufficient  guidance  to  CBT  developers  is 
that  lessons  are  frequently  changed.  One  of  the  most  frustrating  situations  for  a  courseware 
developer  is  to  be  told  to  do  things  (me  way  one  day,  and  another  way  the  next  This  is 
particularly  a  problem  after  a  CBT  lesson  has  been  programmed  because  changes  which  affect 
graphics  and  branching  can  be  extremely  time-consuming  to  execute.  However,  major  changes 
(x:curring  late  in  CBT  development  are  characteristic  of  most  CBT  efforts  inside  the  Air  Force. 
Respondents  reported  that  even  after  lessons  were  on-line,  changes  were  made  to  lesson 
objectives  (17.6%),  graphics  (49.3%),  sequence  of  instrucdon  (40.1%),  branching  (48.6%),  and 
lesson  content  (69.7%).^  In  addition,  a  number  of  interviewees  made  comments  to  the  effect 
that  "developers  should  not  be  subject  to  the  whims  of  several  reviewers,"  and  that  development 
took  longer  than  expected  because  "supervisors  kept  making  changes  to  lesson  format  and  style." 
As  one  respondent  remarked,  "With  no  consistent  directions  from  the  top,  and  constant  changes, 
no  one  is  sure  what's  going  on  from  day  to  day.  What  you're  working  on  today  may  be  cancelled 
tomorrow."  These  comments  indicate  that  two-way  communication  of  the  imm(>diate  goals  of 
CBT  projects  can  be  improved. 

Another  indicator  that  Air  Force  CBT  developers'  comments  are  representative  is  that 
CBT  planning  is  not  fully  conducted  prior  to  beginning  some  CBT  projects.  Proper  planning 
provides  the  roadmap  for  everything  that  follows.  If  planning  steps  are  omitted,  then  they  will 
need  to  be  made  up  for  later  in  the  project  F(H'  example,  if  objectives  are  not  developed  or  are 
not  current,  then  changes  to  lessons  will  be  necessary  at  some  point  perhaps  as  late  as  after  a 
lesson  fails  validation.  Or,  if  key  personnel  are  not  convinced  via  a  media  analysis  that  CBT  is 
the  appropriate  medium  there  will  always  be  uncertainty,  which  may  translate  into  lack  of 
support,  regarding  the  wisdom  of  that  choice. 

It  is  difficult  for  CBT  managers  to  provide  sufficient  guidance  for  CBT  developers 
because  relatively  little  guidance  exists  for  the  managers  themselves.  Additionally,  there  may  be 
a  tendency  for  managers  and  developers  not  to  ask  what  is  available.  ATC  Pamphlet  50-4,  The 
CAI  Decision  Handbook,  provides  helpful  guidance  for  managers.  It  !q>pears  that  managers  need 
more  guidance  about  the  roles  and  duties  of  C3T  personnel,  how  to  estimate  cost  and 
development  time,  and  what  effective  CBT  should  look  like.  One  manager  noted  in  an 
interview,  that  he  is  required  to  know:  "how  long  does  it  take  to  develop  CBT  courseware,  ratio 
of  development,  timekeeping,  reccml  sharing,  and  additional  duties."  Another  manager  said, 
"Problem  areas  were  not  addressed  by  the  guidance  that  currently  exists,  e.g.,  problems  created 
by  ignorance  of  CBT  but  adherence  to  authcnity  (do  it  because  I  said  so);  who  should  run  the 


*  These  are  discussed  in  greater  detail  in  the  section  "Efficiency  of  CBT  Development." 
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program;  lack  of  regulatory  guidance  or  parameters  for  CBT  projects;  lack  of  manpower 
formulas/standards  and  development  times." 

Some  organizations  have  internal  guidelines.  For  example,  TAC  has  developed  its  own 
CBT  format  guide  and  the  3480  TCHTW  at  Goodfellow  AFB  has  developed  a  CBT  Style  Guide. 
The  Armstrong  Laboratory  has  developed  the  Guidelines  for  CBT  Planning,  Selection,  and 
Implementation,  which  will  be  published  as  volume  7  of  Air  Force  Pamphlet  50-68,  during  the 
second  quarter  of  FY92. 

Coordination  of  Team  Efforts 

In  most  cases,  CBT  projects  are  team  efforts  bringing  together  personnel  with  diverse 
skills,  such  as  instructional  developers,  subject  matter  experts,  graphic  artists,  video  experts,  and 
programmers.  Team  members  are  often  unfamiliar  with  CBT,  or  have  not  worked  together 
before.  A  substantial  proportion  of  survey  participants  indicated  that  team  members  were  not 
aware  of  their  roles  and  responsibilities  (22.2%);  a  formal  chain  of  authority  did  not  exist  within 
the  CBT  project  team  (21.4%);  activities  of  each  team  member  were  not  coordinated  throughout 
the  project  (25.8%);  and  communication  between  each  of  the  team  members  was  not  clear  and 
effective  (27.4%).^  These  data  indicate  that  coordination  of  CBT  was  not  always  present  at  the 
team  level  and  reinforce  the  notion  that  CBT  managers  would  benefit  from  additional  guidance 
or  training. 

Meeting  Deadlines 

Delays  are  commonplace  in  Air  Force  CBT  development  Many  respondents  reported 
that  there  were  delays  in  storyboard  production  (35.1%),  on-line  lesson  production  (44.4%),  and 
implementation  (28.2%). These  findings  are  likely  manifestations  of  inadequate  planning, 
personnel  turnover,  inexperience  and  insufficient  training.  More  discussion  of  the  reasons  for 
delays  will  appear  in  the  section  "Efficiency  of  CBT  Development  Process." 

Familiarity  with  CBT  Technology 

Many  survey  respondents  said  that  their  managers  were  not  computer  literate,  were  not 
aware  of  the  benefits  of  CBT,  and  were  resistant  to  new  technology.  As  one  remarked,  "People 
in  the  highor  decision-making  positions  don't  really  understand  what's  involved  in  the 
development  of  [CBT]  projects.  They  have  trouble  understanding  why  it  takes  so  long  to 
develop  a  project"  At  the  team  level,  program  or  contract  managers  need  to  evaluate  the 
products  pr^uced  during  development  and  to  advise  developers  about  how  to  produce  better 
courseware.  Ideally,  these  managers  were  once  CBT  developers  themselves;  if  not,  simply 
attending  a  brief  course  prior  to  beginning  a  project  might  be  helpM,  but  appears  to  be 
insufficient  to  provide  all  the  prerequisite  knowledge  that  CBT  managers  need.  Figure  8 
displays  the  reported  level  of  experience  and/or  training  in  CBT  instructional  design  techniques 
for  CBT  program  and  contract  managers.  Large  percentages  of  program  managers  (39%)  and 


’  Data  fiom  survey  questions  59, 61, 63  &  64. 
10  Data  from  survey  questions  1 15, 152  &  171. 
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contract  managers  (38.1%)  lack  practical  CBT  experience.  Considerably  smaller  numbers  of 
program  managers  (20.2%)  and  contract  managers  (9.5%)  lack  training  and  even  smaller 
percentages  of  program  managers  (11.7%)  and  of  contract  managers  (4.8%),  lack  both 
experience  and  training.  According  to  development  team  members'  comments,  training  is  not 
adequate  to  prepare  managers  for  CBT. 


Figure  8.  Managers  Lacking  CBT  Experience,  Training,  or  Both 
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Expressed  as  percentages  of  respondents  reporting  that  the  given 
job  categories  on  their  development  team  lacked  experience, 
lacked  training,  or  lacked  both.  Data  from  survey  questions  58, 
65, 67, 68  &  70. 


Summary 

Some  of  the  comments  made  by  respondents  blame  management  for  problems  in  Air 
Force  CBT  programs.  Other  data  in  this  survey  reinforce  their  position.  Specific  causes  of  CBT 
management  problems  are  addressed  in  this  report,  althou^  our  data  cannot  be  used  to 
investigate  Air  Force  CBT  management  practices.  Further  research  is  required  on  management 
problems  in  order  to  improve  the  overall  quality  of  CBT  produced  in-house. 

Preparation  of  C3T  Development  Personnel 

CBT  development  requires  multiple  skills.  To  quote  one  participant,  "Not  just  any 
person  can  frll  a  slot  for  a  courseware  developer."  In  addition  to  subject  matter  expertise,  it 
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requires  knowledge  of  CBT  instructional  design  techniques,  writing  skills,  graphic  design  skills, 
and  some  degree  of  programming  ability.  Although  no  single  developer  needs  to  have  all  of 
these  skills,  it  is  best  if  every  developer  is  familiar  with  more  than  one.  Unfortunately,  a  large 
number  of  Air  Force  CBT  development  persoimel  are  inadequately  prepared  for  CBT 
development  jobs.  Figure  9  shows  that  more  than  half  of  the  personnel  involved  in  CBT 
development  in  each  job  category  are  inexperienced,  and  a  lar^  ptxtion  of  instructional 
developers  (23.1%),  course  instructors  (34.7%),  and  SMEs  (27.5%)  lack  both  training  and 
experience.  The  lack  of  experienced  and/tH*  trained  persormel  can  be  attributed  to  four  primary 
reasons. 

o  High  turnover 

o  Inability  to  obtain  qualified  CBT  persormel 

o  Lack  of  planning  for  training 

o  Insufficient  training 


Figure  9.  Development  Persormel  Lacking  CBT  Experience,  Training,  or  Both 
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Note.  Expressed  as  percentages  of  respondents  reporting  that  the  given 
job  categories  on  their  development  team  lacked  experience, 
lacked  training,  or  lacked  both. '  Data  from  survey  questions  58, 
65, 67, 68  &  70. 
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Turnover  of  CBT  personnel 


Air  Force  organizations  continually  lose  qualified  CBT  personnel  because  of  high 
turnover.  Figure  10  shows  the  average  percent  of  turnover  reported  for  the  aggregate  and  by 
command.  Turnover  is  defined  here  as  the  percent  of  personnel  leaving  a  CBT  development 
over  one  year.  The  average  percent  of  tumovo*  reported  fw  the  sample  was  26.6%.  So,  on 
average,  CBT  development  teams  lost  at  least  one  out  of  four  personnel  during  a  CBT 
development  effort.  As  most  project  managers  know,  high  turnover  on  any  project  can 
adversely  affect  schedules  and  pr^uct  quality.  The  problem  is  compounded  when  high  turnover 
is  coupled  with  the  inability  to  replace  experienced  personnel. 


Figure  10.  Mean  Percentage  of  Turnover  by  Command 


Note.  Expressed  in  the  percentage  of  respondents  from  the  given 
command  who  answered  survey  questions  89  &  90,  and  Personnel 
Profile  Questioimaire. 


Respondents  reported  several  reasons  for  turnover,  including  transfer  of  personnel  to 
other  projects,  transfer  to  another  duty  station,  or  leaving  the  Air  Force.  Figure  11  shows,  by 
command,  the  reasons  for  turnover  reported  by  respondents.  All  commands  except  ATC 
reported  changing  projects  and  teassigiunent  of  personnel  as  the  most  frequent  causes  of  CBT 
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project  team  personnel  turnover.  Respondents  from  ATC  cited  reassignment  and  leaving  the  Air 
Force  as  the  most  common  reasons  for  turnover. 


Figure  11.  Reasons  for  Personnel  Turnover  by  Command 
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Note.  Expressed  as  percentages  of  respondents  who  reported  that 
personnel  were  lost  due  to  turnover.  The  three  possible  leasmis 
were  not  mutually  exclusive,  i.e.,  respondents  could  check  off  all 
reasons  that  applied.  Data  from  survey  questions  89  &  91. 


It  is  noteworthy  that  changing  projects  is  cited  so  frequently  as  a  reason  for  turnover, 
especially  by  SAC  (63.6%),  MAC  (85.7%),  and  commands  (63.6%)  other  than  ATC.  Cenainly, 
the  high  degree  of  turnover  must  affect  a  CBT  project  Figure  12  shows  that  the  most  frequently 
observed  effects  were  that  development  time  increased  (72.1%)  and  retraining  was  required 
(56.6%).  Figure  6^^  shows  that  turnover  was  a  major  reason  why  the  budget  did  not  accurately 
anticipate  actual  costs  (42.1%).  It  was  already  shown  to  contribute  to  project  overruns.  This 
survey  did  not  quantify  the  effect  of  turnover  on  courseware  quality  but  the  siuvey  data  do  show 
that  25.7%  of  the  respondents  said  turnover  does  adversely  affect  (3T  quality. 


The  large  percentage  of  ATC  participants  influences  the  totals  for  "All  Commands." 
This  figure  is  fully  explained  in  the  section  "CBT  Planning  Procedures." 
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ATC  reported  more  CBT  personnel  being  reassigned  (81.5%)  and  leaving  the  Air  Force 
(55.4%)  than  other  commands.  In  contrast,  ATC  has  a  much  lower  percentage  of  personnel 
being  reassigned  to  another  project  (32.3%)  than  the  other  commands.  It  appears  almost  twice 
as  likely  that  personnel  assigned  to  an  ATC  (TBT  project  will  continue  with  it  contrasted  to  other 
commands.  It  is  a  sound  strategy  whether  based  on  conscious  management  policy  or  occurring 
by  coincidence. 


Figure  12.  Typical  Effects  of  Personnel  Turnover 
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Note.  Expressed  as  percentages  of  respondents  who  reported  that 
development  personnel  were  lost  due  to  turnover.  Data  from 
survey  questions  89  &  92. 


Obtaining  Qualified  CBT  ] 
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The  turnover  of  experienced  CBT  personnel  would  not  be  as  great  a  problem  if 
organizations  could  more  easily  replace  them  with  qualified  CBT  personnel.  Two  commonly 
cited  reasons  for  not  using  personnel  with  CBT  experience  for  CBT  development  were  that 
qualified  personnel  were  not  available  (55.6%),  or  they  were  working  on  other  projects  (15%). 
Another  reason  why  some  Air  Force  organizations  are  unable  to  obtain  qualified  CBT  personnel 
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may  be  the  assumption  that  CBT  personnel  must  also  be  subject  matter  experts  (SMEs).  Many 
organizations  believe  that  it  is  easier  for  an  SME  to  pick  up  CBT  skills  tlum  it  is  to  try  to  teach 
subject  matter  knowledge  to  an  experienced  CBT  developer.  In  contrast,  many  contractor  CBT 
developers  tend  to  rely  on  instructional  designers  and  developers  as  the  core  of  the  CBT 
development  team,  with  the  SMEs  changing  to  meet  individual  project  requirements.  The 
respondents  reported  that  experienced  CBT  personnel  were  not  used  because  they  did  not  have 
background  in  the  subject  matter  (29.3%).  Another  12.8%  indicated  that  experienced  developers 
were  not  required,  which  may  again  indicate  a  belief  that  CBT  experience  is  valued  much  less 
than  subject  matter  knowledge.  Both  types  of  skills  are  necessary,  whether  they  are  found  in  a 
single  developer  or  shared  across  the  development  team.^’ 

The  main  reason  it  is  difficult  to  obtain  qualified  personnel  is  probably  that  there  is 
currently  no  way  to  identify  for  assignment  Air  Force  personnel  with  CBT  skills.  One  of  the 
most  commonly  mention»l  problems  in  the  survey  discussion  section  (24.7%),  and  the 
interviews  (25.2%)  was  the  lack  of  an  Air  Force  Specialty  Cjode  (AFSQ  or  special  experience 
identifier  (SEI)  for  personnel  with  CBT  experience.  The  researchers  and  Armstrong  Laboratory 
scientists  also  had  difficulty  finding  experienced  CBT  personnel  for  this  survey.  The  approach 
was  to  contact  organizations  that  were  currently  developing  or  implementing  CBT  programs. 
This  method  may  also  be  used  by  Air  Force  organizations  attempting  to  obtain  experienced  CBT 
developers. 

CBT  Developer  Training 

Since  it  appears  that  over  half  of  Air  Force  CBT  development  personnel  are 
inexperienced,  it  seems  especially  important  that  they  receive  adequate  training.  However,  Air 
Force  planners  often  neglect  to  anticipate  and  provide  for  this  need.  Various  reasons  were  given 
for  not  providing  CBT  training.  In  some  cases  training  was  not  provided  because  there  was  no 
time  for  it  (37.8%),  training  was  not  available  (25.4%),  or  training  cost  was  not  budgeted 
(13.4%).  Each  reason  suggests  a  lack  of  proper  planning  for  CBT.  Oily  36.6%  of  the 
respondents  reported  that  training  was  not  necessary  b^use  personnel  were  experienced.*^ 

Figures  13  and  14  show  the  content  of  training  received  by  developers.  The  only  notable  weak 
areas  appear  to  be  in  CMl  (53.7%),  software  structure  (48.8%),  and  software  modification 
(33.1%).  These  will  be  discussed  later  in  this  reporL 


*3  Data  from  survey  questions  65  &  66. 
Data  from  survey  questions  68  &  69. 
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Figure  13.  Content  of  CBT  Training 
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Note.  Expressed  as  percentages  of  respondents  reporting  that  CBT  Training  was 
provided.  Data  from  survey  questions  68  &  71. 


Although  35.4%  of  respondents  reported  that  CBT  training  w<is  not  available,  there  are 
numerous  sources  of  CBT  training  which  might  have  been  tapped.'^  Nevertheless,  personnel  are 
being  put  into  positions  of  responsibility  on  CBT  projects  without  training;  not  surprisingly, 
there  are  numerous  comments. 

o  "Personnel  have  been  tasked  to  develop  courses  without  adequate  training, 
expertise,  and  guidance." 

o  "Most  of  my  training  has  been  self-taught" 
o  "The  more  training,  the  better.  Developers  can't  get  enough." 


ATC  runs  a  Computer-Based  Instruction  (CBI)  Designers  course  at  Sheppard  AFB, 
(J3AZR75(XX)-(X)3)  and  an  IVD  Designers  course  (E3AZR75000-001)  and  an  IVD  Managers 
course  (E3A2317S000-000)  at  Keesler  AFB.  In  ad^tion,  there  are  numerous  CBT  authoring 
courses  conducted  by  the  various  vendors  of  authoring  system  software. 
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Figure  14.  Content  of  Authoring  Software  Training 
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Note.  Expressed  as  percentages  of  respondents  reporting  that  authoring 
software  training  was  provided.  Data  ftom  survey  questions  1 19 
&120. 


Summary 

The  Air  Force  is  faced  with  the  problem  of  having  inexperienced  personnel  work  on 
CBT  projects.  While  inexperience  can  be  corrected  by  training,  it  appears  that  CBT  project 
managers  do  not  provide  training  for  their  personnel,  or  do  not  know  where  to  get  the  required 
training.  As  long  as  the  Air  Force  is  faced  with  the  problem  of  identifying  experienced  CBT 
personnel,  training  new  developers  will  be  a  constant  requirement 

Effective  Use  of  CBT 

Every  instructional  medium  has  specific  c^abilities  associated  with  it  CBT  includes  the 
ability  to  present  graphics,  animation,  audio,  video  and  simulation  on  the  same  platform;  to 
interact  with  students;  to  provide  self-paced  instruction;  and  to  track  student  progress  and 
prescribe  remediation  or  enrichment  as  necessary.  Effective  use  of  CBT  involves  taking  full 
advantage  of  these  capabilities  whenever  it  is  instrucdonally  valid  to  do  so  (Alessi  and  Trollip, 
1985).  The  data  suggest  that  Air  Force  CBT  personnel  are  not  utilizing  CBT  to  its  full  potential. 
Although  they  are  using  some  CBT  features,  they  are  not  yet  taking  advantage  of  all  the 
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capabilities  of  the  medium.  This  may  be  due  to  many  factors,  including  a  lack  of  understanding 
about  what  goes  into  the  selection  of  a  CBT  system. 

CBT  System  Selection 

Systematic  selection  of  CBT  hardware  and  authoring  software  is  critical  to  the  effective 
use  of  CBT  because  it  directly  affects  the  kinds  of  lessons  that  can  be  developed.  Perhaps  the 
most  important  consideration  in  selecting  CBT  hardware  and  software  is  whether  or  not  the 
hardware  and  software  selected  allow  developers  to  create  instructionally  effective  lessons  that 
meet  training  requirements.  For  example,  if  there  are  requirements  for  the  students  to  learn 
switch  actions  using  a  control  panel,  the  need  f(»’  a  touch  screen  should  be  a  consideration  during 
hardware  selection  in  order  to  facilitate  learning  and  enhance  the  transfer  of  skills  from  the 
training  environment  to  the  actual  job.  The  data  indicate:  1)  hardware  seems  to  be  selected 
based  on  which  software  runs  on  it;  2)  thoe  seems  to  be  a  lack  of  awareness  that  training 
requirements  should  be  the  primary  consideration  in  CBT  system  selection;  3)  CMI  is  only  taken 
into  account  slightly  more  than  half  of  the  time;  and  4)  student  characteristics  are  infrequently 
considered. 

Figures  15  and  16  show  that  software  requirements  seem  to  drive  the  selection  of 
hardware  (72.3%),  and  that  software  is  selected  based  on  features  other  than  meeting  training 
requirements.  While  three  factors  -  ease  of  use  (78.5%),  presentation  capabilities  of  the 
authoring  system  (75.3%),  and  hardware  requirements  (67.7%)  ->  are  reported  to  be  taken  into 
account  most  often  in  selecting  CBT  software,  there  are  other  factors  considered  by  the  CBT 
system  selection  team.  Certainly,  it  is  important  to  ensure  that  the  CBT  system  can  be  used  by 
developers  with  little  or  no  effort,  and  that  it  is  ctqrable  of  taking  advantage  of  a  wide  variety  of 
presentation  features.  However,  making  the  developer's  job  easier  with  a  user-friendly  CBT 
system  which  does  not  offer  the  single  feature  that  ad^sses  the  most  critical  trairung  objectives 
will  counteract  any  effort  to  develop  quality  training. 


Figuie  IS.  Considerations  in  Hardware  Selection 
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Note.  Expressed  as  percentages  of  respondents  who  reported  that  the 
given  issues  were  considoed  during  the  hardware  selection 
process.  Data  from  survey  questions  49  &  SI. 


When  only  S3.8%  of  the  respondents  report  considering  CMI  requirements  in  selecting  a 
CBT  system  it  may  be  that  CBT  is  being  used  like  other  training  media,  rather  than  as  the  self¬ 
teaching,  self-managing  tool  of  trainers  that  it  was  intended  to  be.  This  might  be  expected  based 
on  the  low  percentage  of  system  administrators  (23.7%)  and  instructors^^  providing  input  to 
software  selection.  However,  both  CMI  and  other  considerations  may  be  secondary  to  the 
higher  authorities  that  ultimately  make  the  C3T  system  decision.  Over  two-thirds^''  of  the 
respondents  indicated  that  such  decisions  were  made  by  higher  authorities. 


The  percentage  of  instructor  input  reported  varies  depending  on  which  specific 
question  is  analyzed.  When  respondents  were  asked  about  the  job  categories  involved  in 
hardware  (24.1%)  and  software  (22.6%)  selection  the  percentages  were  lower  than  when  asked  if 
instructor  input  (36.6%)  was  considered.  In  either  case,  the  percentages  of  input  from  this  group 
are  quite  low. 

Based  on  hardware  selection  (67.5%)  and  software  selection  (66.7%). 
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Figure  16.  Considerations  in  Software  Selection 
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Note.  Expressed  as  percentages  of  respondents  who  reported  that  the 
given  issues  were  considered  during  the  software  selection  process. 
Data  from  survey  questions  52  &  54. 


Finally,  it  seems  that  there  is  litde  effwt  to  match  the  CBT  system  to  the  learning  styles 
of  the  students,  since  student  characteristics  are  infrequently  ctmsidered  (22.6%).  When  these 
pieces  of  information  are  put  together  they  form  a  picture  of  CBT  system  selection  based  on 
characteristics  which  include  ease  of  use  for  the  developer  and,  periiaps,  hoks^  i.e.,  presentation 
capabilities.  This  indicates  the  need  for  more  guidance  for  personnel  who  select  CBT  systems. 

Utilization  of  CBT  Media  Features 

Air  Force  CBT  developers  appear  to  use  CBT  as  if  it  were  a  conventional  training 
medium.  During  the  interviews  many  participants  indicated  that  their  courses  were  converted  to 
CBT  from  other  media  such  as  stand-up  instruction,  workbooks,  or  slide-tape  presentations. 
However,  Air  Force  developers  appear  to  be  using  many  of  the  presentation  modes  supported  by 
CBT;  95.2%  reported  that  graphics  were  develq>ed,  ^.2%  reported  use  of  audio,  and  45.4% 
reported  use  of  still  or  modon  video.  Graphics  were  used  most  frequendy  for  systems  diagrams 
(83.1%)  and  to  display  equipment  (80%).  Video  was  most  often  used  to  teach  idendficadon 
(e.g.,  of  parts  of  an  equipment  panel,  parts  of  a  human  body,  etc.,  84.3%)  and  procedures  (e.g., 
equipment  panel  operadon,  use  of  a  stethoscope,  etc.,  75.7%).  These  applicadons  of  graphics 
and  video  suggest  that  the  capabilides  of  CBT  are  being  used  well.  However,  audio  may  be 


33 


used  less  effectively.  Some  of  the  least  commonly  used  types  of  audio  cues  were  signals 
(30.9%)  and  engine  sounds  (11.1%).  If  CBT  were  being  used  to  simulate  the  job  environment, 
one  would  expect  these  types  of  cues  to  be  used  more  frequently.  Instead,  audio  is  most  often 
used  for  rewards  (45.7%),  which  is  more  typical  of  page  turning  than  high  quality  CBT;  and 
verbalizations  (65.4%),  i.e.,  spoken  directions  or  explanations,  which  are  an  excellent 
application  of  audio  for  students  with  lower  reading  aptitudes.  While  there  appears  to  be  wide 
sc^e  use  of  the  various  CBT  features,  it  may  still  be  necessary  for  Air  Force  developers  to 
change  their  paradigm  from  converting  traditional  instruction  to  CBT,  to  making  full  creative 
use  of  CBT.** 

Evaluation  Features.  Unlike  their  ability  to  take  advantage  of  the  graphics,  video  and 
other  presentation  features  of  CBT,  Air  Force  developers  have  not  yet  begun  to  take  full 
advantage  of  the  evaluation  power  of  CBT.  Almost  all  Air  Force  developers  utilize  embedded 
questions  (93.8%)  and  feedback  (98.7%)  in  lessmis.*’  However,  as  with  presentation  features, 
developers  are  not  using  evaluatitm  features  in  a  marmer  which  allows  them  to  test  student 
performance  in  ways  which  closely  approximate  on-the-job  performance.  Figure  17  shows  the 
types  of  questions  used.  Multiple  choice  (96.6%)  and  true/false  (87.2%)  predominate.  Less 
commonly  used  are  questions  which  require  indicating  an  area  of  the  screen  by  moving  the 
cursor  or  touching  a  pointer  (49%),  constructed  response  (24.2%)  and  verbal  response  (4%). 
Since  most  Air  Force  CBT  lessons  are  techrtically  oriented  and  graphics  depicting  equipment  are 
frequently  used,  one  would  expea  "area  of  the  screen"  type  questions  to  be  used  more 
frequently.  The  widespread  use  of  multiple  chcnce  and  true^alse  questions  suggests  that  Air 
Force  CBT  developers  are  oriented  toward  paper-and-pencil  type  testing,  even  though  the 
computer  is  capable  of  more  complex  testing  modes  (Horowitz,  1988).  Once  again,  this 
indicates  the  ne^  for  developers  to  tmak  out  of  the  old  instructional  paradigm. 


I*  Data  from  survey  questions  130, 132, 137, 138, 140  &  141. 
Data  from  survey  questions  125  &  128. 
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Figure  17.  Types  of  Questions  Used 
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Note.  Expressed  as  percentages  of  respondents  who  reported  that 
questions  were  embedded  within  the  lessons.  Data  from  survey 
questions  126  &  127. 


CMI  Features.  Computer-Mana^  Instruction  (CMI)  can  be  a  powerful  tool  when  used 
effectively.  The  computer  can  be  used  to  keep  track  of  various  indicators  of  student 
performance,  to  prescribe  remediation  or  enrichment  as  needed,  and  to  free  the  instructor  to 
concentrate  on  student  problems.  However,  CMI  is  not  being  used  effectively  in  the  Air  Force 
where  it  is  used  as  a  computerized  gradebot^  rather  than  as  a  tool  to  help  prescribe  training. 
Figure  18  shows  the  types  of  CMI  data  collected.  The  items  most  commonly  collected  were  test 
scores  (92.6%)  and  student  demographic  data  (91.7%).  Less  frequently  collected  were  time  in 
lesson  (72.7%),  embedded  question  scores  (64.5%),  answer  analysis  (59.5%),  and  number  of 
attempts  per  question  (47.9%);  however,  these  ctq>abilities  c(^  be  employed  for  more 
sophisticated  teaching  functions,  such  as  assigning  stu^nts  to  particular  lessons  or  courses  based 
on  their  performance. 
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Figure  18.  Types  of  CMI  Data  Collected 


Note.  Expressed  as  percentages  of  respcndents  reporting  that  the 
computer  was  utilized  for  CMI.  Data  hrcHn  survey  questions  122 


&123. 


The  number  of  respondents  reporting  use  of  the  computer  for  CMI  was  lower  than 
expected  (72.5%).  This  may  be  because  many  devel(q>ers  are  unaware  of  the  benefits  of  CMI. 
Figure  19  shows  that  firequently  cited  reasons  for  not  utilizing  the  computer  for  CMI  include 
"Data  collection  was  not  required"  (21.7%);  "Programmers  were  not  trained  to  develop  CMI" 
(21.7%);  "Information  collected  by  the  CMI  was  not  necessary"  (15.2%);  and  "The  instructor 
was  responsible  for  data  collection"  (15.2%).  All  of  these  reasons  suggest  that  many  Air  Force 
CBT  developers  are  still  thinking  in  a  conventional  training  mode. 

One  interesting  finding  cmiceming  CMI  is  "Authraing  software  did  not  provide  CMI 
capability"  (16.7%).  If  student  scheduling  and  management  ctq)abilities  are  desired,  CMI 
requirements  must  be  defined  before  the  authming  software  is  selected.  As  one  respondent 
pointed  out,  'Td  like  to  stress  the  impatance  of  identifying  all  user  expectations  of  CMI  early 
on,  and  ensuring  CMI  is  addressed  thoroughly  throughout  the  acquisition  and  acceptance 
phases."  Another  respondent  complained  about  the  authoring  system  selected:  "No  meaningful 
data  can  be  extracted  from  the  system  for  input  into  CAMS.  If  the  student  can't  take  the  course 
and  get  credit  in  CAMS,  he  won't  bother."  If  the  requirement  for  compatibility  with  CAMS  had 
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been  identified  early,  it  could  have  been  considered  in  the  software  selection  process.  However, 
only  53.8%  of  respondents^  indicated  that  CMI  features  were  considered  in  the  authoring 
software  selection  process. 


Figure  19.  Reasons  Why  CMI  Data  Not  Collected 


Note.  Expressed  as  percentages  of  respondents  reporting  that  the 
computer  was  not  utilized  for  CMI.  Data  from  survey  questitms 
122  &  124. 


On-Line  Help  Features.  A  usefril  feature  of  CBT  is  its  ability  to  provide  readily 
accessible  reference  information  to  students  as  they  go  through  lessons.  Simply  by  pressing  a 
function  key  or  selecting  an  item  from  a  pull-down  menu,  etc.,  a  student  can  access  information, 
such  as  a  technical  order  (T.O.)  references,  glossaries,  diagrams,  and  charts.  Ideally,  on-line 
help  should  function  as  the  CBT  equivalent  of  a  job  aid.  In  many  cases,  on-line  help  is  not 
provided  to  Air  Force  trainees.  Only  74.4%  of  participants  indicated  that  on-line  help  was 
developed  for  students.  The  reasons  for  not  providing  on-line  help  suggest  a  lack  of 
understanding  of  how  to  use  this  feature.  The  most  commonly  cited  reason  was  "Such  assistance 
was  not  required"  (60%).  It  would  be  interesting  to  know  if  the  students  felt  the  same  way  about 


^  See  Figure  16,  "Considerations  in  Software  Selection." 


37 


this  assistance.  Other  reasons  included  "The  authoring  software  did  not  suppon  help  screens" 
(22.5%)  and  "Programmers  were  not  trained  to  develop  on-line  documentadon"  (20%).^^  Figure 
20  shows  the  types  of  on-line  help  offered.  The  most  commonly  available  types  of  on-line  help 
were  glossaries  (63.4%)  and  T.O.  or  manual  references  (42.9%).  Less  available  types  included 
lesson  maps  (39.3%),  diagrams  (35.7%).  and  charts  or  tables  (34.8%).  One  way  of  determining 
the  kind  of  on-line  help  required  and  how  often  it  should  be  available  is  to  carefully  assess 
student  characteristics  and  match  these  with  the  training  requirements.  Although  83.8%  of  the 
respondents  reported  that  student  characteristics  were  used  to  determine  training  modes, ^  use  of 
on-line  help  indicates  that  such  information  is  not  being  utilized  fully. 

Ability  to  Offer  Self-Paced  Instruction.  CBT  can  provide  one-on-one  instruction  to 
students  at  any  time  and  in  almost  any  place.  While  CBT  does  not  presently,  and  may  never, 
completely  eliminate  the  need  for  human  assistance,  it  can  either  reduce  the  number  of 
instructors  required  per  class,  or  eliminate  the  need  for  formally  scheduled  classes  (Orlansky  and 
String,  1979).  The  majority  of  survey  participants  were  from  formal  schools,  where  we  should 
expect  CBT  to  be  applied  to  eliminate  traiiung  bottlenecks,  reduce  training  time  or  eliminate  the 
need  for  a  dedicate  instructor.  CBT  should  facilitate  the  migration  of  training  from  formal 
schools  to  job-site.  The  data  suggest  the  Air  Force  is  not  yet  taking  full  advantage  of  this 
capability.  Most  respondents  (80.4%)  indicated  that  an  instructor  was  available  to  assist  students 
as  they  went  through  the  CBT  courseware,  suggesting  that  CBT  is  being  used  in  some  kind  of 
formal  training  environment,  perhaps  in  a  classroom  setting.  Additionally,  class  sizes  are  fairly 
small,^^  which  suggests  that  instructors  are  still  teaching  and  that  instruction  is  probably  lock 
step,  thereby  negating  the  power  of  CBT  as  a  caching  medium. 


21  Data  from  survey  questions  143  &  144. 

22  Data  from  survey  questions  14  &  16. 

23  70.2%  of  the  classes  have  12  or  fewer  students. 
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Figure  20.  Types  of  On-Line  Help  Offered 


Note.  Expressed  as  percentages  of  resptmdents  whd  reported  that  on-line 
help  was  developed  to  assist  the  students  as  they  progressed 
through  the  lessons.  Data  from  survey  questions  143  &  145. 


Summary 

These  data  indicate  that  more  work  is  required  to  change  the  traditional  orientation  of  Air 
Force  CBT  usms  and  developers  so  they  can  t^  full  advantage  of  CBTs  power.  Both  CBT 
developers  and  faculty  must  reorient  their  way  of  thinking  away  from  classrooms  and  traditional 
instructor-guided  leanting  environment  to  a  student-centered  environment  that  makes  use  of  all 
the  CMI  features  of  CBT.  This  reorientation  will  require  a  significant  commitment  from  all 
levels  of  Air  Force  leadership. 

Efficiency  of  the  CBT  Development  Process 

Survey  data  indicate  Air  Force  CBT  courseware  develqmient  appears  to  be  fiequently 
delayed,  over  budget,  and  inefficient  This  is  expected  considering  the  problem  areas  discussed 
so  far.  Often,  incomplete  planning  leads  to  frequent  changes  during  development  Untrained 
and  inexperienced  managers  and  developers  cannot  be  expected  to  do  things  right  the  first  time 
working  with  such  a  complex  medium. 
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Figure  21.  Reasons  for  Delay  in  Storyboard  Completion 


nsb4 


UmoMr 


444 


ftviewFtaoess 


RsMisians 


NaMrTVaining 

R9C|yiremBits 

ftawf  Design 
SareSards 

NBwODrtert 

FteMroiErts 


0  10  2030405060% 


Note.  Expressed  as  percentages  of  resptmdents  who  reported  that 
storyboards  were  not  completed  rni  time.  Data  from  survey 
questions  115  &116. 


Development  Schedule 

Air  Force  CBT  development  is  subject  to  firequent  delays  in  the  production  of 
storyboards  (35.1%)  and  the  on-line  production  of  lessons  (44.4%).  These  delays  probably 
reflect  the  effects  of  incomplete  planning,  high  personnel  turnover,  and  a  lack  of  experienced  or 
trained  personnel  —  all  issues  that  have  been  previously  addressed.  Figures  21  and  22  show  the 
reported  reasons  for  storyboard  and  on-line  lesson  delays,  respectively.  As  expected,  mmover 
and  inexperience  are  two  of  the  primary  reasons.  Other  results  described  in  this  report  point  out 
that  a  lack  of  planning  and  ptq)aration  account  for  several  problems.  Both  storyboar^ng  and 
on-line  lesson  production  are  delayed  most  often  by  revisions.  Although  some  revision  is 
expected  during  the  storyboard  phase  of  CBT  production,  it  can  be  controlled  by  properly 
defining  the  training  requirements  and  lesson  objectives  during  Job/Task  Analysis.  Certainly, 
revisions  during  the  on-line  production  phase  (55.1%)  are  not  well  timed.  These  data  coupled 
with  other  reasons  cited  -  "Training  requirements  changed"  (20.4%  for  storyboards;  30.4%  for 
on-line  lessons),  and  "Design  standards  changed"  (20.4%  for  storyboards;  23.2%  for  on-line 
lessons)  --  suggest  inefficiency  in  Air  Force  CBT  development  For  the  most  pan.  Air  Force 
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organizations  that  were  able  to  complete  on-line  lesson  production  on  time  conformed  to  the 
standard  industry  practice  of  using  CBT  team  members  who  were  experienced  and  trained  in 
CBT  design  and  development  (p<OS). 


Figure  22.  Reasons  for  Delay  in  On-line  Lesson  Completion 
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Note.  Expressed  as  percentages  of  respondents  who  reported  that  lesson 
production  was  not  completed  tm  time.  Data  from  survey 
questions  152  &  153. 


Thc.CBT  Review  Process 

Typically,  CBT  is  reviewed  many  times  in  the  process  of  producing  a  lesson.  Reviews 
usually  occur  during  the  develqmient  of  lesson  specifications,  during  stcHyboarding,  when 
lessons  are  first  put  on-line,  and  during  validation.  Unfortunately,  lessons  attract  the  most 
attention  when  they  are  put  on-line.  Many  people  who  should  have  been  initially  involved  in 
reviewing  lesson  specifications  or  interpreting  stOTyboards  get  their  first  look  at  a  lesson  when  it 
is  put  on-line.  This  is  also  the  least  efficient  point  during  CBT  development  to  make  major 
changes.  Changes  are  difficult  and  time-consuming  to  make  and  disrupt  lesson  and  CBT 
development 
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Ideally,  major  decisions  involving  course  structure,  lesson  objectives,  lesson  format,  and 
lesson  content  should  be  made  prior  to  courseware  development  In  addition,  many  CBT 
contractors  develop  a  sample  lesson  to  show  everyone  who  is  involved  in  CBT  review  what  to 
expect.  During  the  storyboard  review  stage,  corrections  can  address  the  content  of  the  lesson, 
method  of  presentation,  graphics,  and  so  on— but  these  changes  should  never  extend  to  changing 
the  lesson's  objective.  On-line  lesson  review  should  require  even  fewer  changes.  Most  changes 
should  have  been  identified  and  agreed  upon  at  the  storyboard  stage,  although  Air  Force  CBT 
development  does  not  seem  to  follow  this  approach. 

Storyboard  review.  Of  respondents^  who  utilized  storyboards  for  CBT  development, 
92.1%  reported  that  these  storyboards  were  reviewed  by  SMEs  or  course  instructors  for 
acceptability.  Figure  23  shows  the  types  of  revisions  resulting  from  storyboard  review;  20.7%  of 
respondents  reported  that  objectives  had  to  be  changed.  This  is  a  very  high  percentage 
considering  how  late  this  review  takes  place  in  CBT  development  Two  other  items  that  suggest 
a  lack  of  front-end  planning  arc  the  need  to  revise  the  instructional  sequence  (50.4%)  and 
revisions  of  lesson  format  (47.9%).  Large  numbers  of  these  changes  should  not  be  necessary  at 
this  stage,  and  reflect  the  inexperience  and  lack  of  training  of  CBT  development  personnel 
surveyed.  It  should  be  noted  that  some  emphasis  is  being  placed  on  student  interactivity 
(46.3%).  This  could  mean  that  many  lessons  start  out  as  page  turners  because  of  the  developers' 
inexperience  with  CBT  capabilities,  or  because  the  developers  are  converting  existing  lessons  to 
CBT,  as  though  they  were  developing  a  text  lesson. 


Of  respondents  83.2%  reported  that  storyboards  were  used.  Data  from  survey  question 
105  &  107. 


42 


Figure  23.  Types  of  Revisions  Resulting  from  Storyboard  Review 
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Note.  Expressed  as  percentages  of  respondents  who  reported  that 
storyboard  revisions  were  deemed  necessary  after  review.  Data 
from  survey  questions  107, 1 10  &  1 1 1. 


On-line  review.  On-line  review  is  normally  used  to  identify  those  few  discrepancies  that 
remain  in  a  lesson  after  the  storyboard  has  been  approved.  On-line  review  was  conducted  by 
91.9%  of  the  respondents.  Various  changes  tir  lessons  based  on  this  review  are  shown  in  Figure 
24.  The  number  of  respondents  reporting  changes  to  objectives  (17.5%)  at  this  stage  in  CBT 
development  is  startling.  Changes  occurring  during  this  stage  reflect  the  same  lack  of  front-end 
work  previously  discussed,  including  changes  to  instructional  sequence  (40.1%)  and  lesson 
format  (47.2%).  In  spite  of  the  attention  paid  to  interactivity  during  storyboard  review,  other 
significant  revisions  (38%)  were  required.  It  appears  that  most  of  the  problems  associated  with 
CBT  lessons  were  detected  and  corrected  during  on-line  review. 
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Use  of  audit  trail  procedures.  One  common  source  of  inefficiency  during  review  is  that 
changes  to  lessons  are  made  repeatedly,  because  there  is  no  record  of  the  rationale  for  the 
change.  Survey  data  show  that  28.1%  of  Air  Fence  respondents  did  not  keep  records  of  changes 
made  to  storyboards,  and  30.3%  did  not  keep  records  for  on-line  lesson  changes.  Conscientious 
use  of  audit  procedures  would  encourage  efficiency  in  Air  Force  CBT  development^^ 


Figure  24.  Types  of  Revisions  Resulting  from  On-Line  Review 
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Note.  Expressed  as  percentages  of  respondents  who  reported  that 
revisions  were  required  after  the  on-line  review.  Data  from  survey 
questions  147, 1S0&  ISl. 


Summary 

The  inefficiency  reflected  in  CBT  development  is  probably  related  to  the  lack  of 
experience  and  training  of  many  personnel.  These  problems  might  also  be  attributed  to  a  lack  of 
specific,  validated  guidance  on  how  to  develop  CBT  technologies. 


^  Data  from  survey  questions  1 12  &  154. 
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Standardization 


In  any  large  orgamzation  such  as  the  Air  Force,  standardization  is  important  because  it 
facilitates  information  dissemination  and  exchange.  There  are  two  types  of  CBT 
standardization:  (1)  standardization  of  CBT  configuration  (hardware  and  software)  and  (2) 
courseware  standardization.  Based  on  our  data,  the  Air  Force  can  improve  in  both  areas. 

CBT  Configuration 

There  is  no  cunent  standard  CBT  configuration  of  hardware  and  software  for  the  Air 
Force.  One  advantage  of  establishing  a  standard  is  to  facilitate  sharing  courseware  among 
organizations.  Although  the  survey  did  not  specifically  investigate  hardware  and  software 
standardization,  it  was  frequently  mentioned  in  the  discussion  section  and  during  interviews. 
The  use  of  different  authoring  packages  is  a  particular  problem;  in  one  case,  as  many  as  six 
different  authoring  software  packages  were  being  used  at  the  same  base.  Some  developers  like 
to  choose  among  the  capabilities  of  a  variety  of  authoring  software  packages  and  feel  a  standard 
authoring  package  would  be  too  restrictive.  While  we  do  not  advocate  a  single  standard  for  CBT 
authoring  software  to  serve  all  Air  Force  users  because  of  their  differing  training  requirements, 
some  degree  of  standardization  might  be  helpfuL  One  representative  conunent  was:  "The  Air 
Force  has  been  acquiring  CBT  training  and  systems  for  years,  and  there  is  no  standard  for  this 
CBT.  This  results  in  years  spent  on  development  of  CBT  and  it  is  for  a  sole  user  with  no  chance 
of  transportability  between  systems.  This  means  we  'reinvent  the  wheel'  or  design  a  lesson  over 
and  over  again  for  each  system.  Also,  there  is  no  clearinghouse  for  lessons  developed  or 
designed  that  could  be  used  if  there  was  tranSpartability."  The  general  opinion  is  that  some 
standardization  of  CBT  configuration  is  desirable. 

Courseware  Standardization 

There  is  also  need  for  standardization  at  the  individual  course  level.  This  assures 
agreement  among  everyone  involved  at  each  im>jea  stage,  and  gives  a  polished  appearance  to 
lessons.  There  are  several  "tools"  for  courseware  standardization:  format  (style)  guides,  sample 
lessons,  flowcharts,  and  storyboards. 

Format  guides  are  collections  of  rules  and  standards  set  by  an  organization  for 
programming  lessons.  A  typical  format  guide  addresses  tc^ics  like  lesson  structure,  screen 
design,  colors,  use  of  questions  and  feedback,  Inanching,  titles,  and  menus  (Eckstein,  1990; 
436th  STS,  1991).  Fmmat  guides  provide  all  lessons  with  the  same  "look,"  which  aids  student 
learning.  Format  guides  also  facilitate  review  and  revision  of  lessons  by  people  other  than  the 
original  lesson  writer  or  programmer.  While  many  respondents  (77.3%)  reported  that  their 
organizations  utilized  format  guides,  the  number  is  less  than  expected  if  good  CBT  development 
practices  were  being  consistently  followed.^ 

Sample  lessons  complement  standardization  by  demonstrating  how  a  lesson  might  look. 
Because  many  people  have  difficulty  visualizing  how  a  lesson  will  look  based  only  on  a  format 


Data  from  survey  question  95. 
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guide  description,  sample  lessons  are  essential  tools  in  CBT  development  However,  only  52.1% 
of  respondents  reported  that  sample  lessons  were  used.  Perhaps  this  may  be  a  contributing 
reason  why  respondents  made  revisions  in  lesson  format  (47.2%)  when  lessons  were  on-line.^ 

Flowcharts  depict  the  sequence  and  types  (e.g.,  information,  question,  diagram)  of 
screens  which  will  be  used  in  a  lesson  and  the  way  the  lesson  will  flow  from  one  screen  to 
another.  Developers  often  use  flowcharts  to  help  organize  their  thoughts;  but,  flowcharts  are 
most  useful  as  tools  to  assure  agreement  among  key  development  personnel  so  that  corrections  to 
lesson  sequence,  branching,  etc.^can  be  made  before  storyboarding  begins.  Howchans  are  used 
by  77%  of  respondents.  Although  a  relatively  large  number  of  respondents  use  flowchans  to 
standardize  lessons,  50.4%  also  reported  changes  in  instructional  sequence  due  to  storyboard 
review  and  40.1%  due  to  on-line  review.  Either  the  developers  ignor^  the  flowcharts,  ^d  not 
understand  their  use  or  the  flowcharts  were  incorrect^ 

Summary 

Standardization  of  CBT  systems  and  development  approaches  can  lead  to  more  efficient 
use  of  CBT  in  the  Air  Force.  However,  using  existing  standardization  procedures  can  improve 
CBT  if  everyone  understands  the  procedures  and  how  to  take  advantage  of  them.  This  also 
supports  the  need  to  provide  additional  training  for  CBT  developers. 

Validation 

The  primary  purpose  of  validation  is  to  determine  whether  or  not  the  courseware 
achieves  the  specified  standards  by  effectively  teaching  objectives  to  the  target  audience. 
Although  93.1%  of  respondents  reported  that  courseware  was  successfully  validated,  it  is 
unlikely  that  rigorous  validation  takes  place  in  many  Air  Force  CBT  efforts  based  on  types  of 
validation  activities  performed  and  the  organization  of  validation  plans. 

Types  of  Validation  Activities 

Diere  are  several  types  of  validation  activities  that  are  conducted  including  formative  and 
summative  evaluation  (Gagne  and  Briggs,  1979;  Dick  and  Carey,  1985).  Common  procedures 
are  to  conduct  a  formative  evaluation  with  instructors,  SMEs  and  individual  stu^nts,  then 
conduct  a  summative  evaluation  with  representatives  of  the  student  population  first  in  a  small 
group,  and  finally,  if  time  and  resources  allow,  with  a  large  group  in  a  setting  as  much  like  the 
intended  training  environment  as  possible.  The  data  shown  in  Figure  25  suggest  that  many  Air 
Force  CBT  developers  fail  to  v^date  courseware  with  members  of  the  target  audience.  It 
appears  that  validation,  if  attempted,  usually  consists  of  instructor  or  SME  review  (64%)  and 
tryout  lessons  with  individual  students  (71.1%),  i.e.,  formative  evaluation.  Far  fewer  conduct 
summative  evaluation  (small  group  try-outs  53.5%,  large  group  tryouts  35.1%),  and  even  fewer 
attempt  to  relate  the  training  to  job  performance  (28.9%).  lliese  numbers  indicate  that  some 
limited  validation  activities  are  taldng  place. 


^  Data  from  survey  question  98. 
^  Data  from  survey  question  102. 
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Figure  25.  Types  of  Validation  Activities  Performed 
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Note.  Expressed  as  percentages  of  respondents  who  were  involved  in  the 
validation  of  CBT  lessons.  Data  from  survey  questions  158  & 
160. 


Validation  Procedures 

Validation,  like  most  complex  CBT  procedures,  is  normally  guided  by  a  plan.  The  plan 
identifies  data  to  collect,  procedures  to  foUow,  and  standards  of  evaluation  to  achieve.  The 
quality  of  the  courseware,  as  well  as  the  experience  of  the  development  team  can  be  revealed  by 
examining  the  validation  plan.  Many  respondents  (68.8%)  reported  using  a  validation  plan.  A 
large  percentage  of  Air  Force  organizations  probably  did  not  have  a  plan  to  guide  courseware 
validation.  Even  those  organizatimis  that  developed  a  validation  plan  (hd  not  seem  to  understand 
validation.  Figme  26  shows  the  typical  items  specified  in  validation  plans.  For  example,  24.7% 
did  not  describe  student  involvement;  37.7%  failed  to  describe  instructor  involvement;  23.4% 
did  not  specify  the  data  collection  requirements  ot  data  collection  procedures  to  follow  (39%),  or 
what  to  do  with  the  data  once  collected  (58.4%).  Perhaps  the  most  telling  item  is  that  few 
respondents  (55.8%)  had  speciEed  the  standards  to  be  achieved  at  the  end  of  the  validation 
phase. 
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Figure  26.  Contents  of  Validation  Plans 
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Note.  Expressed  as  percentages  of  respondents  reporting  the  use  of  a  validation  plan. 
Data  from  survey  questions  161  &  162. 


Summary 

Survey  data  indicate  that  the  Air  Ftnce  is  not  conducting  effective  validation  of  CBT. 
This  tends  to  fit  in  with  the  lack  of  coordinated  planning  for  CBT  ^parent  from  the  survey. 
Without  effective  validation  the  Air  Force  cannot  determine  if  the  courseware  is  achieving  the 
training  objectives,  and  furthermore,  if  the  courseware  and  objectives  match  the  job 
requirements. 


Implementation 

Implementation  of  CBT  in  the  Air  Force  often  appears  to  be  complicated  by  student  and 
instructor  reluctance  to  accept  the  courseware.  Participants  agreed  that  this  was  not  a  serious 
problem  because  eventually  almost  everyone  adjusts  to  the  situation.  The  reasons  for  student 
reluctance  should.be  further  investigated.  Some  reasons  for  instructor  reluctance  can  be 
determined  from  the  data.  In  general,  these  problems  can  be  minimized  if  instructional  staff  and 
students  are  sufficiently  prepared  to  accept  their  new  roles. 
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Figure  27.  Reasons  for  Instructor  Reluctance  to'Use  CBT 


Note.  Expressed  as  percentages  of  respondents  who  reported  that 
instructors  were  reluctant  to  reach  using  CBT.  Data  firom  survey 
questions  178  &  179. 


Instructor  Reluctance  to  Use  CBT 

Instructor  reluctance  to  use  CBT  was  reported  by  64.4%  of  respondents.  The  reasons 
reported  in  Figure  27  suggest  that  instructors  do  not  receive  enough  training  to  overcome  their 
fears  and  misconceptions  about  CBT  prior  to  implementation.  As  one  participant  remarked, 
"One  of  the  biggest  things  that  I  think  is  wrong  is  that  our  instructors  are  not  taught  how  to 
effectively  deal  with  the  CBT  (IVD,  CAI)  environment  Because  of  this  no  matter  how  good  or 
professional  your  product  is,  it  will  never  be  any  good  to  them." 

The  survey  distinguished  between  two  types  of  training  fra-  instructors:  "training  on  the 
system,"  i.e.,  how  to  use  the  system;  and,  "training  in  CBT  instructional  techniques,"  i.e.,  how  to 
use  CBT  effectively  as  an  instructional  tool.  Although  86.5%  of  respondents  reported  that 
instructors  received  training  on  the  system,  only  56.4%  indicated  that  instructors  received 
training  on  CBT  instructional  techniques.^ 


Data  from  survey  questions  180  &  183. 
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The  instructor's  role  in  a  CBT  training  environment  is  often  quite  different  from  their 
role  in  traditional  education  and  training  (Andrews  and  Tiainor,  1987;  Stephenson,  1991). 
Instructors  may  be  reluctant  to  give  up  the  spotlight  One  respondent  indicated  that  former 
platform  instructors  have  the  most  difficult  time  adjusting  to  CBT.  Survey  data  showed  that 
more  instructors  are  reluctant  to  use  (TBT  in  a  formal  environment  thim  in  an  informal 
environment  Perhaps  this  is  because  formal  school  instructors  are  accustomed  to  being  the 
focus  of  attention;  whereas  on-the-job  training  supervisors  are  used  to  an  informal,  over-the- 
shoulder  style  of  helping  the  student 

Student  Reluctance  to  use  CBT 

Many  students  (46.5%)  were  reluctant  to  use  CBT.  Commonly  cited  reasons  include 
"lack  of  foiniliarity  with  the  system"  (74.6%);  "lack  of  keyboarding  skills"  (47.5%);  and 
"concern  about  system  reliability,  e.g.,  answer  judging,  smdent  advancement,  etc."  (20.3%). 
These  problems  could  be  easily  remedied  either  with  proper  student  preparation  for  CBT  or  by 
using  another  input  device  whenever  possible.  Another  reason  for  student  reluctance  was 
"difficulty  reading  information  presented  on  the  computer  screen"  (20.3%).  This  reflects  poor 
CBT  lesson  design  that  could  be  corrected  by  developers  having  more  experience  or  training  in 
screen  design. 

Summary 

Student  aitd  faculty  reluctance  to  use  C3T  may  be  eliminated  with  !^)ecialized  training. 
Faculty  involvement  in  (ZBT  development  as  reviewere  of  the  courseware,  or  as  SMEs  will 
quickly  overcome  their  fears  of  CBT  and  malre  them  feel  part  of  the  team,  rather  than  as 
recipients  of  training  developed  by  someone  else. 

Maintenance 

Problems  with  CBT  hardware  and  software  can  delay  CBT  development  and  disrupt 
implementation.  Fortunately,  most  Air  Force  hardware  and  software  problems  appear  to  be 
relatively  minor.  Many  coidd  be  alleviated  with  better  planning  and  preparation  of  personnel. 
The  sections  below  describe  specific  problems  with  networics,  hardware,  authoring  software,  and 
documentation. 

Network  Problems 

Networks  are  useful  when  several  students  use  the  computer  at  the  same  time  or  when 
other  software,  in  addition  to  CBT,  must  be  shared  by  the  users.  A  significant  proportion  of 
respondents  (58.4%)  are  using  networks  for  their  CBT  systems.  ^ 

Networks  can  reduce  the  cost-per-student  for  software  and  allow  lessons  and  CMI  data  to 
be  stored  in  one  location  rather  than  duplicated  on  several  different  computers  or  on  floppy 


Data  from  survey  question  208. 
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diskettes  (Air  Training  Command^  1989).  However,  network  pafoimance  may  be  degraded  if 
there  are  too  many  uscts  or  the  system  may  simply  run  out  of  stwage  capacity.  Such  problems 
can  be  avoided  or  alleviated  if  factors  such  as  class  size,  training  site,  and  software  requirements 
are  considered  during  the  planning  stage.’^ 

Two  problems  with  networks  were  investigated:  (1)  problems  associated  with  student 
load  exceeding  system  capabilities  (27.3%)  and  (2)  problems  sharing  the  system,  i.e.,  use  of  it 
for  other  types  of  applicadons  in  addition  to  CBT  overloading  the  system  c^radty  (25.7%). 
Figure  28  shows  the  consequences  of  these  problems,  including  degradation  of  system  speed 
(57.9%)  or  storage  (45.5%).  Other  negative  effects  were  that  the  organization  was  not  able  to 
add  more  courseware  (26.3%),  could  not  add  more  student  stations  (21.1%),  or  did  not  have 
enough  memory  for  the  storage  of  data  such  as  student  Hies.  Each  of  these  problems  can  be 
attributed  to  a  lack  of  proper  planning  for  the  CBT  system  during  its  acquisition  phase. 


Figure  28.Types  of  Problems  with  Networked  CBT  Systems 
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Note.  Expressed  as  percentages  of  respondents  who  reported  that  the 
requirements  to  share  the  system  hardware  exceeded  its 
capabilities.  Data  from  survey  questions  21 1  &  212. 


31  Figure  15  verifies  that  class  size  (26.5%)  and  training  site  (31.3%)  were  not  widely 
considered  factors  during  hardware  selection. 
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Hardware  Problems 


Hardware  failures  were  experienced  by  73.9%  of  respondents.  To  some  extent,  hardware 
failures  are  inevitable.  The  main  issue  is  whether  or  not  they  have  a  deleterious  effect  on 
training  operations,  e.g.,  loss  or  contamination  of  data,  inability  to  conduct  training,  etc.,  and 
whether  or  not  the  failure  can  be  handled  by  available  personnel. 

Some  hardware  failures  experienced  by  Air  Force  CBT  organizations  can  be  fairly 
serious.  Figure  29  shows  that  data  were  sometimes  lost  (43.9%)  or  contaminated  (31.7%). 
Ideally,  serious  problems  should  be  handled  either  by  an  on-site  computer  specialist,  i.e.,  a 
system  administrator,  or  by  a  maintenance  contract  with  a  system  vendor  to  minimize  down¬ 
time.  However,  22.6%  of  respondents  reported  that  a  system  admiiustrator  was  not  available, 
and  30.5%  reported  that  hardware  maintenance  contracts  were  not  purchased.^^  These  data 
indicate  that  some  problems  associated  with  hardware  failures  might  be  prevented  by  prior 
planning. 


Figure  29.  Types  of  Hardware  Failures 
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Note.  Expressed  as  percentages  of  respondents  who  reported  that 
hardware  failures  had  been  experienced.  Data  ftom  survey 
questions  213  &  214. 


Data  from  survey  questions  173  &  215. 
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Software  problems  may  be  classified  into  two  categories:  (1)  software  bugs  and  (2) 
inadequate  software  capabilities.  Software  bugs  were  reported  by  53.1%  of  respondents.  Figure 
30  shows  several  typical  authoring  software  problems.  Some  of  these,  such  as  "commands 
inoperable"  (29.4%)  and  "function  keys  inoperable"  (19.6%),  can  potentially  handicap 
development  and  student  ability  to  use  the  system.  The  problem  cited  most  often,  "lessons  'hung 
up'"  (60.8%),  may  be  due  more  to  poor  programming,  Lc.,  inexperienced  developers,  rather  than 
to  problems  with  the  software.  These  types  of  software  problems  are  generally  best  handled  by  a 
system  administrator  or  lesson  developer  in  consultation  with  the  software  manufacturer. 
However,  60.4%  of  respondents  reported  that  authoring  software  maintenance  contracts  were  not 
puTchased.^3 


Figure  30.  Types  of  Software  Problems 


Note.  Expressed  as  percentages  of  respondents  who  reported  that  they 
experienced  problems  with  the  authoring  software.  Data  from 
survey  questions  218  &  219. 


Data  from  survey  question  220. 


Some  developers  cited  the  following  inadequate  software  capabilities. 

o  "The  system  is  extremely  slow  when  four  or  more  people  are  on  line.  Graphics 
do  not  allow  animation.  Graphics  are  'drawn'  on  screen  rather  than  simply 
appearing  in  final  form.  This  is  extremely  distracting  and  subject  to  critiques  by 
99%  of  students." 

o  "System  software  was  not  designed  for  graphics  construction,  thus,  it  is  very 
tedious  to  design  graphics.  Also,  graphics  display  is  slow.  Editing  audio  is 
almost  as  frustrating  as  graphics  design.  It  is  a  long,  drawn-out  process-time- 
consuming  and  tedious.  Audio  delivery,  on  the  other  hand,  is  one  of  the  system's 
strong  suits." 

Once  again,  these  problems  can  probably  be  attributed  to  the  lack  of  an  adequate 
software  selection  process  and  a  general  lack  of  planning  for  CBT. 

Documentation  Problems 

Authoring  software  documentation  is  essential  for  courseware  developers;  87.4%  of 
respondents  indicated  that  they  used  documentation.  However,  good  software  d^umentation  is 
not  always  available;  26.4%  of  respondents  reported  that  the  documentation  provided  by  the 
software  vendor  was  not  sufficient  Documentation  problems  can  be  minimized  by  making  good 
documentation  a  software  selection  criterion.^ 

Summary 

Maintaining  a  CBT  system  in  good  operating  condition  consists  of  more  than  just 
reacting  to  hardware  or  software  problems.  Proactive  planning  by  CBT  managers  during  the 
front-end  analysis;  making  use  of  desired  or  required  features  as  part  of  the  selection  criteria  in  a 
trade-off  analysis  of  CBT  systems;  and,  making  use  of  computer  expertise  for  system  planning 
ensures  adequate  maintenance  and  efficiency  of  CBT  systems. 


IV.  DISCUSSION 

The  goal  of  this  Air  Force  CBT  survey  was  to  identify  issues  with  broad  application  for 
Air  Force  users  and  to  characterize  how  various  organizations  handled  them.  Anyone  involved 
with  the  medium  knows  that  every  CBT  project  encounters  some  problems  during  its  life-cycle. 
By  investigating  these  problems,  improvements  to  Air  Force  CBT  planning,  selection, 
development,  and  implementatimi  proce^hires  can  be  made,  and  ultimately  the  quality  of  training 
throughout  the  Air  Force  can  be  improved. 

This  report  has  outlined  several  significant  issues  which  were  described  by  organizations 
involved  in  CBT,  including: 


^  Data  from  survey  questions  224  Sc  225. 
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o  Planning  for  CBT 

o  CBT  management 

o  Training  of  CBT  development  personnel 

o  Capabilities  of  CBT 

o  CBT  development  procedures 

o  Standardization  of  CBT  systems  and  procedures 

Planning  for  CBT 

This  repon  contains  numerous  examples  of  ineffective  planning  for  CBT.  ISD  principles 
and  procedures  should  be  followed  throughout  CBT  development  The  data  show  that  some  ISD 
steps  have  not  been  performed.^^  Some  survey  data  indicate  that  the  required  steps  may  not  be 
completely  understood,  for  example,  validation.^ 

The  effects  of  incomplete  planning  permeate  the  entire  CBT  development  process.  There 
are  cost  and  schedule  problems  that  can  be  directly  related  to  lack  of  planning  and  inexperience 
(see  Figure  6).  Managers  failed  to  plan  properly  for  the  initial  cost  of  hardware  and  software, 
for  maintenance  costs,  or  to  train  personnel.  They  incorrectly  estimated  the  time  for  CBT 
development,  review  of  lessons,  and  revisions. 

Many  problems  associated  with  ineffective  CBT  planning  can  be  corrected  by  providing 
CBT  managers  and  developers  with  clear  guidance  and  training  on  what  to  do  and  how  to  do  it. 
Several  new  tools  and  documents  will  soon  be  available  to  provide  that  guidance.  The  Air 
Training  Command  is  developing  a  new  Air  Ftnce  pamphlet  to  provide  that  guidance.  Air 
Force  Pamphlet  50-68,  Information  for  Designers  of  Instructional  Systems  is  a  multi-volume 
pamphlet  which  includes  an  interactive  courseware  (ICW)  decision  guide,  an  ICW  developer's 
guide  and  the  Guidelines  for  CBT  Planning.  Selection,  and  Implementation  mentioned  earlier. 
ATC  is  also  developing  the  Traimng  Cost  Estimator  System  (TRACES)  which  will  provide  cost 
estimates  of  training  course  costs,  including  CBT  course  costs.  The  Armstrong  Laboratory  has 
developed  the  Guidelines  for  Transportable  Education  and  Training  (GTET).widch  provides 
procedures  and  cost  and  scheduling  models  fcnr  turning  resident  courses  into  transportable 
courses  of  various  forms,  including  CBT. 

CBT  Management 

Our  survey  respondents  were  dissatisfied  with  CBT  management  in  regard  to  providing 
guidance,  schedi^g  and  coordination  of  team  efforts  and  other  program  resources.  They 


See  Table  3  which  shows  accomplishment  of  CBT  planning.  Of  all  respondents  only 
77.6%  performed  Training  Needs  Assessment,  67.1%  -  Job/Task  Analysis,  83.1%  -  Analysis  of 
Trainee  (Characteristics,  64.1%  -  Facilities  Requirements  Assessment,  50.0%  -  Media  Analysis, 
and  86.9%  Developed  Training  Objectives. 

^  Of  the  68.8%  who  reported  using  Validation  Plans,  very  few  included  vital 
information  necessary  to  conduct  the  step  properly  (see  Figure  26).  Air  Force  validation 
activities  center  around  student  and  instructor  review  of  the  courseware  rather  than  any  formal 
test  of  its  effectiveness. 
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viewed  their  managers  as  inexperienced  (see  Figure  8),  and  tended  to  blame  management  for 
most  CBT  problems.  Many  Air  Force  CBT  managers  are  truly  inexperienced  and  this 
contributes  to  problems  in  planning  CBT.  Even  when  managers  did  plan  for  CBT,  they 
frequently  took  the  wrong  factors  into  consideradon,  such  as  in  hardware  and  software  selection 
(see  Figures  15  &  16).  There  have  been  problems  in  developing  CBT  which  may  also  be 
attributed  to  ineffective  management  It  is  unclear  whether  or  not  delays  in  storyboard 
completion  (Figure  21)  and  on-line  lesson  completion  (Figure  22)  can  be  attributed  to 
inexperienced  managers  or  poor  planning.  Ultimately,  completing  programs  on  schedule  is  a 
management  responsibility.  While  further  research  into  the  problems  of  CBT  management  is 
warranted,  it  appears  that  current  managers  need  additional  training  and  guidance  in  how  to  plan, 
develop  and  implement  CBT. 


Personnel  Factors 

Air  Force  CBT  efforts  are  seriously  affected  by  personnel  factors.  Most  CBT  projects 
start  with  less  than  half  of  the  team  having  CBT  experience  and  many  untrained  instructional 
developers  and  SMEs  (see  Figure  9).  Approximately  one  in  every  four  members  of  the  CBT 
team  leaves  before  the  project  is  completed  (Figure  10).  Identifying  and  keeping  experienced 
personnel  is  a  difficult  task  for  CBT  program  managers.  More  than  half  of  the  respondents 
(55.6%)  reported  that  CBT  experienced  personnel  were  not  available  for  development.  CBT 
managers  should  be  aware  that  even  modestly  ambitious  schedules  cannot  be  met  when 
inexperienced  and  untrained  personnel  form  the  majority  of  the  development  team.  Air  Force 
organizations  are  faced  with  the  decision  of  not  starting  a  CBT  project  or  finding  ways  to 
identify  experienced  personnel,  to  train  mexperieiuxd  personnel  and,  then,  to  retain  them. 
IdentiEcation  of  Air  Force  personnel  with  CBT  experience  will  require  action  from  the 
personnel  system  to  flag  and  track  such  indivichials  for  future  assignments.  Efforts  are  under 
way  to  ad^ss  the  problems  of  lack  of  CBT  training  and  experience.  The  Armstrong 
Laboratory  is  developing  an  Advanced  Instructional  Design  Advisor  (AIDA)  to  provide 
inexperienced  instructional  developers  with  guidelines  for  authoring  computer-based  instruction. 
ATCT  is  woridng  to  provide  as  many  spaces  in  their  existing  C3T  management  and  development 
courses  as  possible.  The  Armstrong  laboratory  is  also  pursuing  a  research  effort  parallel  to  the 
AIDA  mentioned  above.  The  Guided  Approach,  Instructional  Design  Advisor  (GAIDA) 
provides  guidance,  following  Dr.  Robert  Gagne’s  nine  events  of  instruction,  to  inexperienced 
CBT  developers. 


Capabilities  of  CBT 

If  the  Air  Force  were  fully  utilizing  CBT  there  would  be  more  individualized  instruction, 
efficient  use  of  CMI  capabilities,  extensive  use  of  graphics,  video  and  audio  to  simulate  the  job 
environment,  and  a  wide  variety  of  evaluation  techniques  used  to  diagnose  learning  problems 
and  prescribe  assistance.  Respondents  (72.5%)  reported  using  the  computer’s  CMI  capabilities. 
Most  of  this  group  were  using  CMI  to  provide  testing  results  rather  than  to  diagnose  student 
problems  and  provide  tailored  remediation  or  enrichment  (see  Hgure  18).  With  such  a  heavy 
reliance  on  inexperienced  personnel  to  develc^  CBT,  it  will  be  difficult  for  the  Air  Force  to 
substantially  improve  the  situation.  Unfortunately,  the  use  of  SMEs  as  the  primary  source  of 
CBT  development  personnel  tends  to  reinforce  this.  As  long  as  the  Air  Force  experiences  a  lack 


56 


of  qualified  personnel,  it  will  continue  to  undenitilize  CBTs  capabilities.  In  the  meantime,  there 
is  a  real  ne^  for  guidance  to  aid  in  selecting  the  most  appropriate  CBT  system  for  a  given 
application  and  applying  the  most  effective  and  efficient  instructional  design,  development  and 
delivery  strategies.  Whether  this  guidance  takes  the  form  of  a  desktop  reference,  additional 
training,  or  an  automated  expert  system  remains  a  subject  for  further  research. 

CBT  Development  Procedures 

CBT  development  cannot  be  efficient  without  systematic  planning  and  management. 
Inexperienced  management  and  development  personnel  compound  the  problem.  There  were 
frequent  delays  in  storyboards  (35.1%)  and  production  of  lessons  on-line  (44.4%).  Most  delays 
were  due  to  the  need  for  revisions  of  various  types,  including  revising  the  objectives  (17.6%). 
The  need  to  revise  objectives  while  validating  a  course  will  delay  the  development  process  and 
may  result  from  the  inexperience  of  the  pmonnel  developing  the  objectives.  Air  Force 
organizations  planning  to  develop  CBT  should  make  it  a  point  to  become  familiar  with  the  CBT 
courses  available,  guidance  systems  being  transitioned  from  R+D  and  DoD  and  Air  Force 
regulations  governing  the  use  of  CBT  an  other  interactive  technologies. 

Standardization  of  CBT  Systems  and  Procedures 

There  are  two  standardization  issues:  1)  the  standardization  of  hardware,  software  and 
authoring  systems,  and  2)  the  standardization  of  lesson  development  procedures  and  practices. 
Standardization  of  CBT  systems  is  rare.  Over  half  of  the  CBT  systems  are  networked  (58.4%), 
while  the  others  are  stand-alone  applications.  Various  authoring  systems  are  being  used 
throughout  the  Air  Force.  CBT  hardware  and  software  selection  seems  to  be  based  on  the 
characteristics  of  each  other  rather  than  on  a  specification  of  training  system  requirements.^^ 

Standardization  of  lesson  development  is  critical  to  the  production  of  quality  CBT. 
Current  Air  Force  CBT  development  practices  vary.  In  spite  of  using  frnmat  guides  (77.3%), 
sample  lessons  (52.1%),  flowcharts  (77%)  and  storyboard  (83.2%),  significant  revisions  are 
made  when  lessons  are  already  on-line  (55.1%).  Air  Force  development  teams  appear  to  be 
following  standardized  procedures,  but  they  may  not  know  the  real  purpose  for  using  these 
standardization  tools.  A  contributing  factor  may  be  the  inexperience  of  development  personnel. 
Improved  guidance,  training  and  stability  of  the  development  team  should  contribute  to  better 
courseware  development  practices. 


Problem  Causes 

Data  concerning  these  issues  have  been  presented  throughout  this  report  The  problems 
have  various  causes  that  stem  frxnn  lack  of  clear  and  useable  guidance  at  the  basic  level. 
Specifically,  respondents  felt  that  there  is  no  Air  Force  implementation  policy  for  CBT  from  the 


Software  requirements  were  considered  by  72.3%  of  respondents  in  selecting 
hardware,  and  hardware  requirements  were  considered  by  67.7%  of  respondents  in  selecting 
software.  Compare  these  figures  to  the  50.6%  of  respondents  who  considered  training 
requirements  when  selecting  hardware,  and  36.6%  when  selecting  software. 
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highest  levels  of  command  down  through  the  various  major  commands  to  the  organizations 
implementing  CBT.  This  is  particularly  true  for  CBT  planning,  which  most  organizations  fail  to 
do  well.  While  most  organizations  attempt  to  plan,  few  accomplish  all  the  required  activities. 
Organizations  tend  to  omit  steps  in  the  planning  process  out  of  ignorance  that  an  activity  should 
be  performed,  rather  than  as  the  result  of  some  conscious  effort  to  do  it  their  own  way. 
Procedures  followed  in  one  organization  are  not  necessarily  the  same  as  those  followed  by 
another  organization.  In  many  instances  commands  and  organizations  develop  their  own  policy 
and  guidance  for  CBT.  While  AFP  50-58  provides  excellent  guidance  to  all  organizations 
developing  training,  individual  organizations  can  tailor  the  procedures  to  fit  their  unique 
requirements.  The  same  kind  of  guidance  provided  in  AFP  50-58  needs  to  be  developed  for  Air 
Force  CBT. 

Another  contributing  cause  of  many  of  the  Air  Force’s  CBT  problems  is  lack  of 
experienced  or  trained  managers  and  CBT  developers.  Survey  results  indicate  that  a  majority  of 
Air  Force  development  team  members  and  a  significant  portion  of  the  management  are 
inexperienced  and  that  personnel  problems  affect  CBT  development  In  most  cases,  the  shon- 
term  solution  to  these  problems  is  to  provide  as  much  training  as  possible  to  the  various  team 
members,  especially  the  project  manager.  In  the  long  run,  tracking  experienced  CBT  personnel 
will  help  alleviate  this  problem. 

Finally,  as  with  most  other  training  problems,  an  ounce  of  analysis  is  -worth  a  pound  of 
development.  Most  reported  problems  result  from  a  failure  to  properly  plan  for  various  CBT 
activities.  Most  organizations  make  attempts  to  plan  for  CBT  but  are  unaware  of  what  planning 
activities  need  to  be  performed  or  how  to  perform  them.  Guidance  for  CBT  planning  and 
pplication  is  urgently  needed. 


Recommendations 

While  the  findings  from  this  survey  are  not  conclusive,  they  are  indicative  that  Air  Force 
CBT  is  in  need  of  improvement  Three  specific  courses  of  action  to  improve  CBT  are 
recommended  based  on  ^e  findings  of  this  survey: 

o  Further  research  is  needed  on  the  quality  of  Air  Force  CBT. 

o  Specific  guidance  and  trairting  are  needed  for  managers  and  developers  of  CBT. 

o  Air  Force  persoimel  capable  of  developing  CBT  need  to  be  ident^ed  for  future 
assignments. 

Further  Research  into  CBT  Quality 

This  survey  did  not  exanune  the  quality  of  Air  Force  CBT.  Instead,  it  focused  on  issues 
related  to  CBT  planning,  selectitm  and  implementation.  Students  were  not  surveyed,  nor  was 
courseware  evaluated.  In  spite  of  this,  wherever  the  researchers  went  they  were  asked  to  look  at 
examples  of  CBT.  Many  sites  felt  that  they  were  producing  the  best  courseware  in  the  Air 
Force.  Their  confidence  may  be  based  on  appearance  rather  than  effectiveness.  This  survey 
indicates  that  few  respondents  (28.9%)  attempted  to  validate  their  courseware  with  on-the-job 
performance,  and  fewer  (18.4%)  sought  supervisors'  evaluations  of  the  product.  Further 
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research  is  needed  to  determine  the  quality  of  Air  Force  CBT,  i.e.,  its  effectiveness  and 
efficiency  as  a  training  medium. 

Guidance  and  Training  for  CBT  Managers  and  Developers 

Data  from  this  survey  suggest  several  problems  associated  with  CBT  development 
They  range  from  improper  planning  for  CBT  to  reluctance  on  the  part  of  instructors  to  accept 
CBT.  Both  CBT  managers  and  developers  need  guidance  to  follow  in  planning,  selecting  and 
implementing  the  medium.  This  guidance  can  take  the  form  of  procedural  manuals,  models,  or 
automated  expert  systems  that  assist  personnel  in  accomplishing  some  aspect  of  CBT.  Those 
involved  in  CST  should  receive  clear  guidance  and  training  in  several  areas,  including  the  steps 
to  follow  in  developing  CBT,  the  rationale  fm  each  step,  how  to  perform  each  step  and  the 
typical  problems  which  might  be  encountered.  The  training  should  provide  the  students  with  the 
opportunity  to  practice  those  skills  before  being  asked  to  use  them  on  the  job. 


Identification  of  Air  Force  CBT  1 
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If  the  Air  Force  intends  to  remain  in  the  CBT  business,  a  final  recoiiunendation  to 
improve  Air  Force  CBT  is  to  provide  a  means  within  the  persoimel  system  to  identify  personnel 
with  CBT  skills  for  future  assignment  Personnel  turnover  and  the  lack  of  qualified 
replacements  are  two  of  the  major  contributing  factors  to  problems  in  CBT.  If  qualified  CBT 
personnel  could  be  identified  for  reassignment  to  positions  in  which  their  CBT  ddlls  could  be 
utilized  it  might  produce  a  stabilizing  effect  on  projects  where  turnover  averages  26.6%  and 
approximately  47.4%  of  the  development  team  is  experienced.  If  assignment  policies  do  not 
permit  such  identification  and  utilization  of  Air  Force  personnel  with  CBT  experience,  then  the 
Air  Force  may  wish  to  consider  limiting  CBT  projects  to  specific  organizations  or  to 
development  by  contractor . 
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APPENDIX 


Note:  Numbers  beside  questionnaire  responses  indicate  the  number  of 
survey  participants  selecting  the  response.  In  some  cases,  numbers  may 
add  up  to  more  or  less  than  the  total  expected.  This  is  usually  because  a 
participant  skipped  the  item,  because  more  than  one  item  could  be 
checked,  or  because  the  question  (Hily  appeared  on  one  version  of  the 
survey. 


CBT  PERSONNEL  PROFILE  QUESTIONNAIRE 


The  puipose  of  this  questionnaire  is  to  identify  Air  Force  personnel  who  have  had  experience  in 
some  aspect  of  Computer-Based  Training  (CBT)  in  an  Air  Force  environment  Please  complete 
this  form  as  quickly  and  completely  as  possible  and  return  it  to _ by _ 

Please  identify  yourself. 

Name/Rank: _ _ _ _ 

Position  and  Job  Title:  _ _ 

Base:  _ _ _ _ _ 

Organization: _ _ _ 

Office  Symbol:  _  Phone: _ 

Please  describe  your  CBT  experience.  Indicate  the  extent  of  your  involvement  for  all 
appropriate  items.  Explain  "Other"  in  fulL 

POSITION  TIME  IN  MONTHS 

Cunent/past  manager  of  CBT  program  _ 

Current/past  developer  of  CBT  courseware  _ 

Cuirent/past  instructor  in  CBT  course  _ 

Current/past  CBT  contract  monitor  _ 

Staff  computer  systems  expert  involved  in  CBT  _ 

Other: _ _ 

Please  name  the  CBT  systems  with  which  you  have  been  associated.  If  possible,  specify 
operating  systems,  hardware,  and  software,  as  well  as  the  relevant  courseware. 
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SCN:  90-72 


U.  S.  Air  Force 
Human  Resources  Laboratory 
Computer-Based  Training  (CBT) 
Survey 


PURPOSE; 

The  US.  Air  Force  Human  Resources  Laboratory  (AFHRL),  in  conjunction  with 
their  support  contractor  Mei  Associates,  Inc.,  is  conducting  a  survey  of  Computer-Based 
Training  (CBT)  in  the  Air  Force.  The  ultimate  goal  of  this  instrument  is  to  identify 
problems  associated  with  CBT  and  the  approach  which  various  Air  Force  training 
organizations  ha'/O  taken  to  solve  these  problems. 


YOUR  INVOLVEMENT: 

Your  organization  has  agreed  to  participate  in  the  CBT  Survey.  You  will  be 
expected  to  answer  the  questions  in  this  questionnaire  to  the  best  of  your  ability.  Once  you 
have  completed  the  questionnaire,  you  should  notify  the  individual  within  your  command 
who  has  been  designated  as  the  point  of  contact  CEOQ.  Hie  POC  will  coli^  all  survey 
materials  and  notify  AFHRL.  You  may  also  be  scheduled  for  a  follow  up  interview  after 
the  questionnaire  bias  been  analyzed.  During  this  brief  interview  you  will  be  asked  about 
some  of  the  specific  answers  that  you  have  given  in  the  questionn^. 


[Please  complete  the  following  ircms  regarding  your  name  and  position.] 


Name: _ 

Rank: _ 

Organization: _ 

Organization  Code: _  Phone:. 
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CBT  SURVEY 


Instructions:  Indicate  your  responses  to  the  questions  by  circling  the  answer  in  the  case  of 
YesINo  questions,  filling  in  the  blank  provided  or  by  marking  one  or  more  of  the  options 
presented  in  the  nadnple  choice  type  questions.  Please  note  that  on  most  of  the  multiple  choice 
questions,  you  may  select  one  or  more  options. 

The  questions  in  this  survey  ask  about  the  various  activities  associated  with  CBT  planning, 
selection,  development,  validation,  implementation,  and  maintenance.  If  you  have  participated 
in  more  than  one  CBT  project,  please  answer  these  questions  on  the  basis  of  what  was  usually 
done,  rather  than  on  what  was  done  for  particular  projects. 

If  you  encounter  any  questions  that  are  ambiguous  or  not  understood,  please  make  a  note  of  this 
in  the  margin  adjacent  to  the  question  tweeted  so  that  we  may  discuss  these  with  you  during  the 
interview  process. 

As  you  complete  this  survey,  keep  in  mind  that  THERE  ARE  NO  RIGHT  OR  WRONG  ANSWERS 
to  these  questions.  No  evaluation  will  be  made  about  you,  your  work,  or  this  training  center 
based  on  the  answers  you  provide.  Each  of  your  responses  are  meaningful  and  will  further 
understanding  of  CBT  planning,  selection  and  implementation  issues.  We  sincerely  apprecuae 
your  cooperation. 
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A.  Planning.  Generally,  planning  for  CBT  involves  the  identification  and  analysis  of 
training  requirements  (e.g.,  training  needs  analysis,  or  job  or  task  analysis),  and  the 
definition  of  training  development  guidelines,  allocation  of  resources,  and  system  selection 
(including  analysis  of  user  characteristics,  media  selection,  and  other  facilities  analysis). 

1.  Have  you  been  involved  in  planning  fot  the  development  and/or  implementation  of  CBT? 
192  53 
Yes  No 

(If  1  =  No,  please  go  on  to  Section  B,  (^estion  47.) 


2.  If  1  =  Yes,  what  was  the  nature  of  your  involvement?  (Check  the  appropriate  lesponse/s.) 

Program  Manager 
24  Contract  Manager 
ill  Instructional  Developer 
43  Instructor  (End-user) 

_§2.  Subject  Matter  Expert 

^7  Other _ 


3.  Have  you  been  involved  in  the  aiudysis  phase  of  planning,  Le.,  identification  of  training 
requirements,  job  or  task  analysis,  characterization  of  target  student  population,  media 
andysis,  etc.? 

145  47 

Yes  No 

(If  3  =  No,  please  go  on  to  Question  28.) 


4.  If  3  =  Yes,  was  a  formal  training  needs  assessment  performed  in  the  planning  phase  of 
CBT? 

Ill  32 

Yes  No 
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5.  If  4  =  No,  why  not?  (Check  the  appropriate  response/s.) 


_ I  There  wasn’t  enough  time. 

_ 1  There  were  not  enough  funds. 

_ §.  Personnel  were  not  available  to  conduct  the  analysis. 

— i  Personnel  (subject  matter  experts,  instructional  experts,  etc.)  were  not  available  to 

provide  the  d^  necessary  to  complete  the  analysis. 

_  Training  needs  were  derived  from  current  course  material. 

— ^  The  traimng  needs  assessment  completed  previously  was  current. 

_9  Other _ 


6.  If  4  =  Yes,  was  the  training  needs  assessment  used  to  identify  training  requirements? 


104  8 

Yes  No 


7.  If  6  =  No,  why  not?  (Check  the  apprc^riate  response/s.) 

_ 1  The  analysis  was  incomplete. 

_ L  The  analysis  was  not  accepted  as  valid. 

— 2  The  analysis  contained  out-of-date  information. 

Other  _ 


8.  If  6  =  No,  how  were  the  training  requirements  identified? 

_ I  Developed  fiom  existing  course  objectives. 

_ -  Developed  from  existing  course  materials  and/or  manuals. 

— 2  Training  requirements  identified  previously  were  current  and,  subsequently,  used 

for  the  CBT  project 

_  Other _ 


9.  Was  a  formal  job/task  analysis  performed  as  part  of  the  CBT  planning  process? 
96  47 

Yes  No 
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10.  If  9  =  No,  why  not?  (Check  the  appropriate  response/s.) 

6  There  wasn’t  enough  time. 

2  There  were  not  enough  funds. 

9  Personnel  were  not  available  to  conduct  the  analysis. 

3  Personnel  (subject  matter  experts,  etc.)  were  not  available  to  provide  die  data 
necessary  to  complete  the  analysis. 

26  Jobs  and  tasks  were  derived  from  current  course  material  and/or  manuals. 

The  job/task  analysis  completed  previously  was  current 
11  Other  _ 


11.  If  9  =  Yes,  were  the  job/task  analysis  results  available  for  use  by  the  instructional 
designers  during  the  development  phase  of  the  project? 

87  8 

Yes  No 


12.  If  11  =  No,  why  not?  (Check  the  r^propriate  response/s.) 

_!t_  The  analysis  was  incomplete. 

1  The  analysis  was  not  accepted  as  valid. 

__2.  The  analysis  contained  out-of-date  information. 

3  Other  _ ^ _ 


13.  If  11  s  Yes,  were  these  results  used  by  the  instructional  designers? 
66  5 

Yes  No 


14.  Did  the  characteristics  of  the  prospective  students,  such  as  reading  level,  prerequisite 
skills,  etc.,  play  a  role  in  the  planning  or  fiont-end  analysis  of  the  project? 

118  24 

Yes  No 
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15.  If  14  =  No,  why  not?  (Check  the  appropriate  response/s.) 

_ L  Thwc  wasn’t  enough  time. 

_ 2.  There  were  not  enough  funds. 

_ ±  Personnel  were  not  available  to  conduct  the  analysis. 

_A  Personnel  (subject  matter  experts,  students  etc.)  were  not  available  to  provide  the 
data  necessary  to  complete  Ae  analysis. 

_il  Other  _ 


16.  If  14  =  Yes,  was  this  information  (smdent  characterization)  used  to  determine  training 
modes  (tutorial,  practice,  simulation,  etc.)? 

98  19 

Yes  No 


17.  If  16  =  No,  why  not?  (Check  the  impropriate  response/s.) 

_ L  The  analysis  was  incomplete. 

_ L  The  analysis  was  not  accepted  as  valid. 

_ 2.  The  analysis  contained  out-of-date  information. 

_iL  Training  mode  was  pre-determined. 

_ 4  Other  _ 


18.  Were  facility  requirements  (HVAC  needs,  room  size,  layout,  anti-static  floor  covering, 
etc.)  assessed  in  the  planning  phase? 

91  52 

Yes  No 


19.  If  18  =  No,  why  not?  (Check  the  appropriate  response/s.) 

_ 5  There  wasn’t  enough  time. 

_ i  There  were  not  enough  funds. 

_ I  Personnel  were  not  available  to  conduct  the  planning. 

_ 6  Persmmel  (systems  experts,  electrician,  fire  marshall,  etc.)  were  not  available  to 

provide  the  data  necessary  to  complete  the  planning. 

J1  Other  _ 
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20.  If  18  =  Yes,  were  facilites  requirements  used  to  select  or  prepare  the  training  site  for  the 
CBT  system? 

70  21 

Yes  No 


21.  If  20  =  No,  why  not?  (Check  the  tq)propiiate  response/s.) 

_ 1  The  plan  was  incomplete. 

_ 1  The  plan  was  not  accepted. 

_ 1  The  plan  called  for  facilities  which  were  not  available. 

_ 0  The  plan  was  based  on  out-of-date  information. 

1  s  Other  _ _ 


22.  Was  a  media  analysis  peiformed  in  the  planning  phase  of  the  CBT  project? 

70  70 

Yes  No 


23.  If  22  =  No,  why  not?  (Check  the  tqq)ropriate  lesponse/s.) 

_ 2  There  wasn’t  enough  time. 

_ 2  There  were  not  enough  funds. 

_ 5  Personnel  were  not  available  to  conduct  the  analysis. 

_ 3  Personnel  (subject  matter  experts,  etc.)  were  not  available  to  provide  the  data 

necessary  to  complete  the  planning. 

5A  Media  had  been  predetermined  by  a  higher  authoriQr. 

20  Other _ _ 


24.  If  22  =  Yes,  what  method  or  tool  was  used  to  conduct  the  analysis?  (Check  the 
appropriate  response/s.) 

48  SME  expertise 

31  Paper-based  media  selection  model 
12  Automated  media  selection  model 

Other  _ _ _ _ _ 


25.  Were  training  objectives  developed  for  the  computer-based  instruction? 
126  19 

Yes  No 
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26.  If  25  =  No,  why  not?  (Check  the  appropriate  response/s.) 

_ P  There  wasn’t  enough  time. 

_ p  There  weren’t  enough  funds. 

_ 2  Personnel  were  not  available  to  develop  the  objectives. 

_ Previously  developed  objectives  were  used. 

_ 1  Training  objectives  were  not  required. 

_ 3  Other  _ 


27.  If  25  =  Yes,  how  were  the  objectives  developed?  (Check  the  appropriate  lesponse/s.) 

_ 88  From  the  job/task  analysis 

_ From  the  media  analysis 

_ 59  From  Tech  Orders  or  manuals 

_ 10  By  modifying  previously  written  objectives 

_ 86  Using  SNIE  expertise 

_ 16  To  utilize  equipment  purchased  or  expected 

68  To  meet  training  requirements 

_ 7  Other  _ 


28.  Have  you  been  involved  in  budgeting  for  CBT  systems? 

85  100 

Yes  No 

(If  28  =  No,  please  go  on  to  Question  34.) 


29.  If  28  =  Yes,  what  was  your  role?  (C3ieck  the  appropriate  response/s.) 

_ 18  Provided  input  for  hardware  acquisition 

_ 74  Provided  input  for  software  acquisition 

_ 26  Provided  input  for  facilities  acquisition 

_ 10  Provided  input  regarding  personnel  requirements 

_ 2.1  Provided  input  tegarding  project  completion  time 

_ 15  Provided  input  tegarding  training  requirements  of  the  project  team 

_ 39  Conducted  cost/benefits  analyses 

_ 29  Developed  the  budget 

_ 10  Other  _ 
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30.  Did  the  budget,  as  developed,  accurately  anticipate  actual  costs? 

39  38 

Yes  No 


31.  If  30  *  No,  why  not?  (Check  the  appropriate  response/s.) 

6  Hardware  cost  more  than  expected. 

6  Software  cost  more  than  expected. 

13  It  became  necessary  to  purchase  additional  hardware. 

11  It  became  necessary  to  purchase  additional  software. 

5  Hardware  maintenance  was  not  factored  in. 

7  Software  maintenance  was  not  factored  in. 

6  The  cost  of  training  of  project  team  members  was  not  factored  in. 

16  Additional  training  was  required  due  to  team  member  turnover. 

9  Production  delays  occurred  while  team  members  learned  to  use  the  system. 
21  Development  took  longer  than  expected. 

J_l_  The  review  process  took  longer  than  expected. 

12  Revisions  caused  unexpected  production  delays. 

14  Training  requirements  changed  causing  courseware  revisions. 

14  Other.  _ 


32.  Did  funding  shortfalls  cause  deviations  from  the  project  development  plan? 
39  42 

Yes  No 


33.  If  32  =  Yes,  what  was  the  result?  (Check  the  ^>proptiate  response/s.) 

15  The  scope  of  the  project  (Le.,  die  number  of  lessons)  was  reduced. 

_ 3  The  length  of  individual  lesstxu  was  reduced. 

9  The  interactivity  within  lessons  was  reduced. 

10  The  number  of  gn^hics  was  reduced. 

17  Less  hardware  was  purchased. 

10  Less  expensive  hardware  was  purchased. 

11  Less  software  was  purchased. 

_ 7  Less  expensive  software  was  purchased. 

12  The  project  was  put  on  hold  until  funding  became  available. 

_ a.  Other  _ _ _ _ 
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34.  Have  you  been  involved  in  the  development  of  a  CBT  Statement  of  Work  (SOW)? 

55  130 

Yes  No 

(If  34  s  No,  please  go  on  to  Section  B,  Question  47.) 


35.  If  34  =  Yes,  what  was  your  role?  ((Theck  the  appropriate  response/s.) 

35  Provided  input  regarding  the  scope  of  the  project 
33  Provided  input  regarding  hardware 

35  Provided  input  regarding  software 

36  Provided  input  regarding  the  deliverable  schedule 

35  Provided  input  regarding  the  deliverable  specifications 
Provided  input  regarding  the  review  process 

36  Provided  input  regarding  acceptance  test  procedures 
17  Wrote  the  SOW 

10  Other  _ 


36.  Was  the  SOW  comprehensive  with  respect  to  the  project  scope  (number  and  length  of 
lessons,  level  of  instruction,  level  of  interactivity,  etc.)? 

42  10 

Yes  No 


37.  What  was  included  in  the  description  of  the  scope  of  the  project?  (Check  the  appropriate 
response/s.) 

38  Number  of  lessons 
23  Length  of  lessons 

36  Level  of  instruction  (initial  vs.  advanced) 

_23  Level  of  interactivity  (c.g.,  x  timcs/3  minutes) 

29  Testing  requirements 
25  Validation  plan 

23  Validation  criteria 

_ 4  Other _ _ 

38.  Did  the  SOW  provide  a  schedule  for  deliverables? 

44  7 

Yes  No 
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39.  If  38  Yes,  were  these  deliverable  dates  achieved? 

19  2A 
Yes  No 


40.  If  38  =  No,  why  not?  (Check  the  appropriate  response/s.) 

_ 7  Production  delays  occurred  while  team  members  learned  to  use  the  system. 

19  E)evelopment  took  longer  than  expected. 

_ i  The  review  process  took  longer  than  expected. 

11  Revisions  caused  unexpected  production  delays. 

_ 8  Training  requirements  changed,  causing  courseware  revisions. 

_ 1  Funding  sht^alls  caused  unexpected  delays. 

16  Other  _ 


41.  Were  data  specification  documents  (DIDs)  required  by  the  SOW? 

34  12 

Yes  No 


42.  If  41  =  Yes,  were  these  documents  provided  to  the  developers? 
29  4 

Yes  No 


43.  If  42  =  No,  why  not?  (Check  the  appropriate  lesponse/s.) 

_ Q  The  specs  were  out-of-date. 

_ 0  The  specs  were  not  relevant  to  the  project 

_ 2  The  developer  was  expected  to  obtain  diem. 

_ li  Other  _ 


44.  Was  the  review  process  described  in  the  SOW? 

40  10 

Yes  No 


45.  If  44  s  Yes,  was  this  process  followed  during  development? 
32  9 

Yes  No 
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If  45  =  No,  why  not?  (Check  the  appropriate  response/s.) 

_ L  Development  delays  reduced  review  time  available. 

_ L  Reviewers’  other  duties  prevented  thorough  review. 

_ 1  Personnel  turnover  altered  the  review  process. 

3  The  process  was  modified  to  reflect  changing  project  requirements. 
_Jl  The  process  did  not  work  as  described. 

_ 6.  Other  _ 


B.  Selection.  The  selection  process  involves  the  determination  of  appropriate  hardware  and 
software  given  training  needs. 

47.  Have  you  been  involved  in  the  selection  of  a  CBT  system  (hardware  and  software)? 

109  135 

Yes  No 

(If  47  =  No,  please  go  on  to  Section  C,  Question  56.) 


48.  If  47  =  Yes,  what  was  the  nature  of  your  involvement?  (Please  check  the  appropriate 
response/s.) 

21.  Program  Manager 

Ji  Contract  Manager 

JJ_  Instructional  Developer 
21.  Instructor  (End-user) 

Subject  Matter  Expert 
_3^  Programmer 
_1^  Graphic  Artist 

24  Other _ 


49.  Have  you  been  involved  in  the  selection  of  CBT  hardware? 
83  25 

Yes  No 


SO.  If  49  =  Yes,  who,  including  yourself,  selected  the  hardware?  (Check  the  appropriate 
response/s.) 

21.  End-users  (instractors) 

21.  Systems  experts 

Courseware  developers 
_5h.  Higher  authorities 
21  Contractor 

Other _ _ _ 
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51.  If  49  =  Yes,  which  of  the  following  were  used  to  determine  hardware  requirements? 
(Check  the  appropriate  response/s.) 

Training  requirements  analysis 

_ 28  Student  characteristics  (human  factors  analysis) 

_ 2_2  Class  size 

_L0  Software  requirements  (authoring  language/system,  graphics  software,  etc.) 

__26  Training  site 

__il  End-user/instructor  input 

_16  System  administrator  input 

Other  _ 


52.  Have  you  been  involved  in  the  selection  of  CBT  software  (authoring  language/system, 
graphics  software,  programming  language,  etc.)? 

93  18 

Yes  No 


53.  If  52  =  Yes,  who,  including  yourself,  selected  the  software?  (Check  the  appropriate 
response/s.) 

__2l  End-users  (instructors) 

Systems  experts 
■  SI  (Courseware  developers 
60  Higher  authorities 
_ L?  Contractor 

_ L3  Other  _ 


54.  If  52  =  Yes,  which  of  the  following  were  used  to  determine  software  requirements? 
(Check  the  iq)propriaie  response/s.) 

Training  requirements  arudysis 
_2l  Student  characteristics  (human  factors  analysis) 

C^MI  reporting  requirements 

70  Presentation  (fonts,  gr^)hics,  colors,  etc.)  capabilities 
Hardware  requirements 
_2}  Ease  of  use 
_34  End-user/mstructOT  input 
_22  System  administrator  input 
_55  Developer  input 

Other  _ 
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55.  What  type  of  authoring  software  was  used  to  program  the  courseware?  (Check  the 
appropriate  lesponse/s.) 

_ Programming  language 

_ LI  Authoring  language 

_ Authoring  system 

_ ?  Other _ 
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C.  Development  This  stage  involves  the  design,  production,  and  initial  review  and  revision 
of  courseware. 

56.  Have  you  been  involved  in  the  development  of  CBT? 

211  33 

Yes  No 

(If  56  =  No,  please  go  on  to  Section  D.  Question  158.) 

57.  If  56  =  Yes,  what  was  the  nature  of  your  involvement?  (Check  the  appropriate 
response/s.) 

84  Program  Manager 

_1J  Contract  Manager 

JA6  Instructional  Developer 
_5l  Instructor  (End-user) 

JJJ  Subject  Matter  Expert 
8^  Programmer 
.66  Graphic  Artist 

37  Other _ 


58.  Who  comprised  the  development  team?  (Check  the  appropriate  response/s.) 

JL06  Program  Manager 
__2J  Contract  Manager 
JAi  Instructional  Developer 
_J2  Course  Instructor 
J.42  Subject  Matter  Expert 
_l^  Programmer 
86  Graphic  Artist 
_25  Contract  Development  Team 

34  Other _ 


59.  Were  each  of  the  team  members  aware  of  his/her  role  and  responsibilities  on  the  project? 
133  38 

Yes  No 
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60.  If  59  =  No,  why  were  the  tasks  unclear?  (Check  the  appropriate  response/s.) 

Turnover  of  development  team  was  high. 

Roles  and  responsibilities  changed  frequently. 

Team  member(s)  wore  more  than  "one  hat” 

Team  member(s)  were  never  informed  about  roles  and  duties. 

No  one  seemed  to  know  who  was  responsible  for  what 
This  was  not  important  to  project  completion. 

Other _ 


61.  Did  a  formal  chain  of  authority  exist  within  the  CBT  project  team? 

132  36 

Yes  No 


62.  If  61  =  Yes.  was  this  hierarchy  followed  during  the  design,  development  review  and 
revision  ptxx:esses? 

116  13 
Yes  No 


63.  Were  the  activities  of  each  team  member  well  coordinated  throughout  the  project? 

121  42 

Yes  No 


64.  Was  communication  between  each  of  the  team  members  clear  and  effective? 

119  45 
Yes  No 


65.  Were  each  of  the  CBT  team  members  experienced  in  CBT  design  and  development? 
37  133 

Yes  No 


16 

8^ 


66.  If  65  =  No,  why  weren’t  individuals  with  CBT  instructional  expertise  selected  for  the 
team?  (Check  the  appropriate  response/s.) 

39  They  did  not  have  background  in  subject  matter. 

20  They  were  working  on  other  projects. 

74  They  were  not  available. 

17  Experienced  developers  were  not  required. 

48  Other _ 


67.  If  65  =  No,  which  team  member(s)  did  not  have  experience  designing  and  developing 
computer-based  instruction?  (Check  the  appropriate  response/s.) 

51  Program  Manager 

13  Contract  Manager 

80  Instructional  Developer 

46  Course  Instructor 

_82  Subject  Matter  Expen 

52  Programmer 

39  Graphic  Artist 

_ 6  Contractor  Development  Team 

19  Other _ _ 


68.  Did  each  member  of  the  development  team  receive  training  in  CBT  insmictional  design 
strategies? 

88  82 
Yes  No 


69.  If  68  =  No,  why  was  this  training  not  provided?  (Check  die  appropriate  response/s.) 

11  Training  was  not  budgeted. 

31  There  was  no  time  for  training. 

29  No  instructional  design  training  was  available. 

30  Training  was  not  deemed  necessary  due  to  the  background  and  experience  of  the 
development  staff. 

Other _  . 
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70.  If  68  =  No,  who  did  NOT  receive  instructional  design  training?  (Check  the  appropriate 
response/s.) 

_ 25  Program  Manager 

_ 5  Contract  Manager 

^2  Instructional  Developer 
36  Course  Instructor 

_ y  Subject  Matter  Expert 

29  Programmer 
Graphic  Artist 

_ 2  Contractor  Development  Team 

_ Other _ 


71.  If  68  =  Yes,  what  material  did  the  training  cover?  (Check  the  appropriate  response/s.) 

69  Tutorial  strategies 
61  Remediation  strategies 
76  Question  development 
76  Feedback/prompt  development 
_ Ll  Types  of  interactivity 

_ LA  Other: _ 


72.  Was  the  CBT  developed  in-house? 
178  29 

Yes  No 


73.  If  72  =  No,  why  not?  (Check  the  t^)propriate  response/s.) 

_ LO  Qualified  instnictional  developers  were  ntH  available. 

_ 2  It  was  not  possible  to  train  instructional  developers  to  use  the  selected  system. 

_ ^1  (^alified  SMEs  were  not  available. 

_ 7  (Qualified  programmers  were  not  available. 

_ L7  It  was  more  cost  effective  to  contract  development 

_ LA  Other _ _ _ 


74.  Were  course  instructors  involved  in  the  development  of  the  CBT? 
140  66 

Yes  No 
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75. 


If  74  =  No,  why  not? 


_ 22  They  were  not  available. 

_ L4  They  were  not  qualified  instructional  designers. 

_ 17  They  lacked  experience  in  CBT  design  and  development 

_ n  Other _ 


76.  If  74  =  Yes,  what  was  their  role?  (Check  the  appropriate  response/s.) 

_ ZO  Wrote  lessons 

_ 91  Provided  objectives  input 

170  Provided  content  input 

_ ^2  Provided  input  regarding  student  characteristics 

_ 60  Provided  input  regarding  screen  design 

_ 71  Provided  input  regarding  lesson  structure  (attention,  motivation,  etc.) 

10*1  Reviewed  lesson  specification  documents,  storyboards,  etc.  for  accuracy 
_Lio  Reviewed  on-line  lessons  for  aoniracy 
_ a6  Provided  quality  control 

_ L4  Other _ _ 


77.  Were  subject  matter  experts  (SMEs)  involved  in  the  development  of  the  C3T? 
196  11 

Yes  No 


78.  If  77  =  Yes,  what  was  their  role?  ((3ieck  the  appropriate  response/s.) 

_121  Wrote  lessons 
_L3l  Provided  objectives  input 
Provided  content  input 

_147  Reviewed  lesstm  specification  documents,  storyboards,  etc.  for  accuracy 
_160  Reviewed  on-line  lessons  for  accuracy 

_J4  Other _ 
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79.  If  77  =  No,  what  resource(s)  were  used  to  ensure  content  accuracy?  (Check  the 
appropriate  response/s.) 

_ ?  Technical  orders  or  manuals 

_ 5  Regulations 

_ n  Existing  course  materials 

_ 3  Commercially  available  texts 

_ 3  Other _ 


80.  Did  the  development  staff  coordinate  their  input  to  the  lesson  content  and  structure? 
179  20 

Yes  No 


81.  If  80  =  No,  why  not?  (Check  the  appropriate  response/s.) 

Coordination  was  not  required  by  the  SOW. 

Distance  made  coordinatioa  difficult 
Team  members  worked  for  different  agencies. 

The  developers  were  expected  to  utilize  input  from  many  sources. 
Other  _ 


82.  If  80  s  Yes,  how  was  the  coordination  accomplished?  (C3teck  the  appropriate  response/s.) 

.HP  Face-to-face  meetings  were  hekL 

_ All  input  was  touted  through  a  single  higher  authority. 

_ iP  Tasks  were  divided  between  development  staff. 

_5l  A  set  of  written  procedures  was  followed. 

17  Other  _ _ 


83.  Were  programmers  used  to  author  the  courseware? 
99  104 

Yes  No 
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84.  If  83  =  Yes,  what  was  their  role?  (Check  the  appropriate  response/s.) 

_ 41  Wrote  the  CBT 

_ ^  Assisted  with  system  operation 

_ ^  Trained  developers  in  system  use 

_ 58  Provided  input  regarding  authoring  capabilities 

_ 38  Provided  input  regarding  the  computer  managed  instruction  (CMI) 

_ 49  Provided  input  regarding  graphics  development 

_ 71  Programmed  CBT 

_ 48  Programmed  CMI  functions 

8  Other:  _ 


85.  Was  a  contractor  used? 

80  124 

Yes  No 


86.  If  85  =  Yes,  what  was  the  contractor’s  role?  (C^heck  the  appropriate  response/s.) 

_ 35  Provide  instructional  developers 

_ Provide  subject  matter  expertise 

_ 47  Provide  programming  services 

_ 30  Provide  developer  training 

_ 13  Develop  courseware 

_ Validate  courseware 

_ 36  Install  equipment 

_ 35  Install  software 

_ 10  Other  _ 


87.  If  86  =  Yes,  did  the  contractor  maintain  an  on-site  presence? 
45  32 

Yes  No 


88.  If  87  =  Yes,  what  did  the  contractor  do  on-site?  (Check  the  appropriate  response/s.) 

_ 15  Facilitate  communication  between  the  contractor  and  client 

_ 15  Collect  data 

_ 22  Develq)  courseware 

_ 23  Install  hardware 

_ 25  SupportAnaintain  hardware 

_ 22  Install  software 

_ 27  SupportAnaintain  software 

_ 18  Provide  training 

_ 9  Conduct  validation 

_ 8  Model  courseware  use 

_ 5  Other _ _ _ 


89.  Were  development  personnel  lost  (due  to  turnover)  during  the  development  stage? 
136  67 

Yes  No 


90.  If  89  =  Yes,  what  percentage?  _ %  “  Al.Az 

^  ^  Median  -  40X 

Mode  »  50Z 

91.  If  89  s  Yes,  why  did  personnel  tumovor  occur? 

_ ^8  Ttansfened  to  another  project 

_ 72  PCS’d  to  another  station 

_ 49  Left  the  Air  Force 

_ 17  Other _ _ 


92.  If  89  s  Yes.  what  were  the  ^ical  results?  (Check  the  iqipropiiate  lesponse/s.) 

_ 98  Develqpmoit  time  increased 

_ 16  Development  time  decreased 

_ 26  Courseware  design  requiiemmits  changed 

_ 26  Courseware  scqie  changed 

_ 77  Retraining  was  required 

_ 35  Lesson  quality  decreased 

_ 1 1  Lesson  quality  increased 

_ 22  Other  _ _ _ _ _ 
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93.  Were  you  involved  in  designing  CBT?  (This  stage  involves  the  development  of  format 
or  design  specifications  and  sample  lessons.) 

161  44 

Yes  No 


94. 


95. 


96. 


If  93  =  Yes,  what  was  your  role?  (Check  the  ^)propriate  response/s.) 

__£P  Program  Manager 

_ ?  Contract  Manager 

114  Instructional  Developer 
38  Instructor  (End-user) 

74  Subject  Matter  Expert 
61  Programmer 
87  (3raphic  Artist 

_ L?  Other  _ 


Was  a  format  or  style  guide  used? 

150  44 

Yes  No 


If  95 

s  Yes,  what  was  covered  in  the  guide?  (Check  the  appropriate  rcsponse/s.) 

86 

Course  structure 

33 

Enrichment 

116 

Lesson  structure 

81 

Review 

97 

Objectives  use 

133 

Screen  Design 

115 

Titles 

131 

Color 

123 

Menus 

106 

Spacing 

124 

(^stions 

47 

Content  enhancements 

130 

Feedback 

(e.g^  advance  organizers) 

121 

Branching 

77 

Ictrns 

55 

Regulations 

104 

Gnq)liics 

82 

Langua^  standards 

63 

CMI 

101 

Testing 

14 

Other 
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97.  If  95  =  Yes,  how  was  the  guide  developed?  (Check  the  appropriate  response/s.) 

114  It  was  developed  in-house. 

15  It  was  developed  by  a  contractor. 

51  It  was  developed  t^ugh  trial  and  error. 

An  existing  guide  was  modified. 

13  Other  _ 


98.  Were  sample  lesson/s  developed  to  guide  the  design  process? 

101  93 

Yes  No 


99.  If  98  s  Yes,  what  guided  the  development  of  these  sample  lesson/s?  (Check  the 
qrprtqniate  response/s.) 

16  SOW 

Regulations 

JiS  Fbrmat/style  guide 
Lesson  objectives 

_10  Commercially  available  texts 
^6  Manuals  and/or  tech  orders 
_72  Authoring  software  durabilities 
29  Odier  lessons 
_22  Templates  (boileqrlates) 

14  Other  _ 


100.  If  98  s  Yes,  were  the  sample  lessons  acc^ted  as  models  for  future  lesson  development? 
87  16 

Yes  No 


101.  If  100  s  No,  why  not?  (Check  the  qrprcqmate  lesponse/s.) 

_ 3  Requiiements  (SOW,  regulatkxis,  etc.)  were  not  met 

_ {t  Design  standards  (ftmnat/s^le  guide,  templates,  etc.)  woe  not  followed. 

_ 2  Lesson  ctmtent  was  inaccurate. 

_ §  Requirements  changed. 

_ 8  Design  standards  changed. 

_JP  Other  _ 
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102.  Were  flow  charts  developed  to  illustrate  lesson  progression? 

154  46 

Yes  No 


103.  If  102  s  Yes,  were  the  flow  charts  accepted  as  guides  ftv  future  lesson  development? 
131  20 

Yes  No 


104.  If  103  =  No.  why  not?  (Check  the  impropriate  response/s.) 

Q  Requirements  (SOW,  regulations,  etc.)  were  not  met. 

5  Design  standards  (foimat/style  guide,  templates,  etc.)  were  not  reflected  in  the 
charts. 

Chart  sequence  was  inaccurate. 

9  Requirements  changed. 

7  Design  standards  changed. 

12  Other  _ 


105.  Were  storyboards  developed  for  the  CBT  lessons? 

164  33 
Yes  No 


106.  If  105  s  Yes,  what  guided  storyboard  ^velopment?  (Check  the  appropriate  response/s.) 

sow 

Regulations 

_3§  Format/styic  guide 
Lesson  objectives 

_1A  Commercially  available  texts 
42  Manuals  and/or  tech  mders 
65  Software  ciq»bilities 
46  Current  traditional  lessons 
38  Sample  lessons 
30  Templates  (boileiplates) 

20  Other  _ 


107.  If  105  s  Yes,  were  the  storyboards  reviewed  by  SMEs  and/or  instructors  for  acceptability? 

151  13 

Yes  No 
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108.  If  107  =  No,  why  not?  (Check  the  appropriate  response/s.) 

_ 1  Review  was  not  required  by  the  SOW. 

_ ^  There  was  not  sufficient  time  fw  storyboard  review. 

_ §  Qualified  SMEs  and/or  instructors  were  not  available. 

_ 6  Storyboard  review  was  not  budgeted. 

_ §  Other  _ 


109.  If  107  s  Yes,  were  the  reviewers  trained  in  instructional  design  strategies? 
71  46 

Yes  No 


110.  If  107  =  Yes,  were  revisions  usually  required? 
121  13 

Yes  No 


111.  If  110  s  Yes,  what  was  the  general  nature  of  diese  revisions?  (Qieck  the  qrpropriate 
response/s.) 

Objective  development 
Sequence  of  instruction 
Content 
__58  Format 

_£!  Grammatica]/spelling4)unctuaticm  mrors 
_54  Gnq)hics 
Interactivity 

Advance  organizers  (titles,  bolding,  etc.) 

_li  Other  _ ^ _ 


112.  Were  records  kept  of  changes  made  to  lessons  following  review? 
123  48  74 

Yes  No  N/A 


113.  If  storyboards  were  developed,  were  they  used  for  on-line  development? 
148  11  86 

Yes  No  N/A 
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1 14.  If  113  =  No,  why  not?  (Check  the  t^propriate  lesponse/s.) 

_ p  Requirements  (SOW,  regulations,  etc.)  were  not  met 

_ 1  Design  standards  (format/style  guide,  templates,  etc.)  were  not  reflected  in  the 

stoiyboards. 

_ 1  Content  was  inaccurate. 

_ p  Training  requirements  changed. 

_ 3  Design  standards  changed. 

_ 4  Content  requirements  chan^d. 

_ P  Required  corrections  were  not  made. 

_ 4  Oth^  _ 


1 15.  Were  the  storyboards  completed  on  time? 
100  54 

Yes  No 


1 16.  If  115  s  No,  why  not?  (Check  the  ^propriate  response/s.) 

24  Personnel  turnover  caused  delays. 

23  The  review  process  took  longo*  than  expected. 

Revisions  caused  une]q)ected  delays. 

.ii  Training  requirements  chan^d,  causing  storyboard  revisions. 

11  Design  standards  changed. 

_14  Content  requirements  changed. 

^6  Other  _ 


117.  Were  you  involved  in  the  on-line  develqnnent  of  the  CBT? 
166  37 

Yes  No 


(If  117  s  No,  please  go  on  to  Section  D,  (^estion  158.) 


118.  If  117  s  Yes,  what  role  did  you  play?  (Check  the  appropriate  response/s.) 


56 

Program  Manager 

81 

Subject  Matter  Expert 

_ ? 

Contract  Manager 

74 

Programmer 

120 

Instructional  Developer 

56 

Graphic  Artist 

40 

InstTJCtor  (End-user) 

26 

Other 
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119.  Was  authming  software  training  provided? 
121  42 

Yes  No 


120.  If  119  =  Yes,  what  was  covered  in  the  training?  (Check  the  appropriate  response/s.) 

116  Text  entiy/editing 
116  Branching 
_26  Interactivity 
Gnq)hics 
65  CMI 

_96  On-line  documentation  (menus,  help  screens,  command/option  lines) 

59  Software  structure 

How  to  modify  the  software 

13  Other  _ 


121.  If  119  =  No,  why  not?  (Check  the  tyrptopriate  response/s.) 

_ 7  Training  was  too  expulsive. 

_ ?  Hme  was  not  available. 

_ 6  Personnel  were  not  available. 

_27  Programmers  were  expected  to  learn  software  on  their  own. 

_ 7  Programmers  already  knew  the  software. 

_ 12  Other  _ 


122.  Did  you  utilize  the  computer  for  CMI  (data  collection  and  student  management)  functions, 
as  well  as  lesson  delivery? 

121  46 

Yes  No 


123.  If  122  s  Yes,  what  data  was  collected?  (Check  the  appropriate  response/s.) 

.Ill  Student  data  (name,  job  title,  etc.) 

112  Test  scenes 
_88  Unie  in  lesstm 

Embetkied  question  scores 
Number  of  attempts  per  question 
_Z2  Answer  aiudysis 

_ 13  Other  • _ 
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124.  If  122  =  No,  why  not? 

_ l_  Authoring  software  did  not  provide  CMI  capability. 

10  Data  collection  was  not  required. 

_ L  Infonnation  collected  by  the  CMI  was  not  necessary. 

10  Progranimers  were  not  trained  to  develop  CML 
7  The  instructor  was  responsible  fen*  data  collection. 

17  Other  _ 


125.  Were  questions  (Le.,  practice  questions)  embedded  within  lessons? 
152  10 

Yes  No 


126.  If  12S  SI  Yes,  did  the  types  of  embedded  questions  vary? 
149  3 

Yes  No 


127.  If  126  s  Yes,  what  types  of  questions  were  used?  (Check  the  jq)propriate  response/s.) 

130  Tnie/false 
144  Multiple  choice 
94  Hll-in 

36  Constructed  resptHise 

73  Areas  of  screen  (move  the  cursor  or  touch  a  pointer  to  select  an  element  on  the 
screen.) 

_ 1  Verbal  tespewse 

16  Other  _ 


128.  Was  feedback  for  embedded  questions  available? 

156  2 

Yes  No 
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129.  If  128  s  Yes,  what  type  of  feedback  was  available?  (Check  the  appropriate  response/s.) 

32  Implied  (question  is  repeated  when  the  student’s  answer  is  wrong) 
QnrectAncorrect  only 

132  ConectAncorrect  wift  explanation/elaboration 
73  Response-specitic  (feedback  varies  with  student’s  answer) 

88  CorrectAncorxect  with  branching  to  remedial  or  advanced  instruction 
Response-specific  with  branching  to  remedial  or  advanced  instruction 
_ 5  Other  _ 


130.  Were  gnq)hics  developed? 
160  8 
Yes  No 


131.  If  130  s  No,  why  not?  (Check  the  tq)propriate  response/s.) 

_ 5  Gr{q)hics  were  not  required  in  the  courseware. 

_ i  Hme  did  not  permit  the  development  of  graphics. 

_ 2  Programmers  were  not  trained  to  develop  graphics. 

_ 1  Available  graphic  fidelity  did  not  meet  trairting  requirements. 

_ 2  Other  _ 


132.  If  130  s  Yes,  what  graphics  were  develq)ed?  (CSieck  the  iq>propriate  response/s.) 

86  Icons  (graphic  symbols  used  as  advance  organizers) 

103  Charts 
_90  Tables 
Diagrams 
_71  Miqts 
128  Equipment 
88  Human  figures 
106  Animation 

22  Other  _ 
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133.  If  130  =  Yes,  what  were  their  chanu;teristics?  (Check  the  appropriate  response/s.) 

The  graphics  were  useful. 

The  graphics  added  interest  to  the  courseware. 

105  The  graphics  were  necessary  to  achieving  the  objective. 

_ 6  The  graphics  were  inaccurate. 

-15  The  graphics  were  too  detailed. 

^11  The  graphics  did  not  contain  enough  detaiL 
The  graphics  were  hard  to  read. 


134.  If  graphics  were  developed,  was  the  authoring  software  used? 
114  39 

Yes  No 


135.  If  134  «  No,  why  not?  (Check  the  appropriate  response/s.) 

.28  Another  software  package  had  more  capabilities. 

— 2  Another  software  package  had  more  "clip  art” 

Another  software  package  was  more  familiar  to  die  programmers. 
_ 1  (3then _ 


136.  If  134  =  Yes,  did  the  ease  of  develqrment  and  the  curabilities  of  the  authoring  software’s 
gn^hics  package  meet  the  project’s  requirements? 

67  29 

Yes  No 


137.  Was  audio  included  in  the  CBT? 
81  87 

Yes  No 
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138.  If  137  s  Yes,  what  type  of  audio  cues  were  developed?  (Check  the  appropriate 
response/s.) 

_2L  Rewards  (bells,  beeps,  tunes) 

25  Signals 
Music 

_ 2.  Engines 

Verbalizations  (commands,  explanations,  questions,  etc.) 

_ 0  StHlgS 

19  othcn  _ 


139.  If  137  =  No,  why  not?  (Check  the  ^)ptopiiate  response/s.) 

Audio  cues  were  not  required  in  the  courseware. 

31  Audio  fidelity  was  not  a .  ailable. 

_ i  Time  did  not  permit  programming  audio  cues. 

Programmers  were  not  trained  to  develop  audio  cues. 
_19  Other  _ 


140.  Did  the  CBT  utilize  still  or  motion  video  images  (Le.,  interactive  video)? 
69  83 

Yes  No 


141.  If  140  s  Yes,  what  was  the  video  used  to  teach?  (Check  the  apprc^niate  respmises/s.) 

Idendficatitm  (e.g.,  parts  of  an  equipment  panel,  parts  of  die  human  body,  etc.) 
Hi  Procedures  (e.g.,  equipment  panel  operation,  use  of  a  stethoscqie) 

Hi  Interpersonal  skills  (e.g.,  dedsion-niaking,  crew  coofdinatkm) 

_ i  Communication  skills  (e.g.,  effective  command  issuance,  foreign  language) 

_ 8  Other  _ 


142.  If  140  s  No,  why  not?  (Check  the  qipropriate  response/s.) 

_29  Video  was  not  required  in  the  courseware. 

_5i  l^deo  capability  was  not  available. 

_I2  Time  did  not  permit  programming  video  sequences. 

17  Programmen  were  not  trained  to  incorporate  video  sequences. 
_ 8  Other _ : _ 
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143.  Was  on-line  help  (T.O.’s,  glossaries,  lesson  miq>s,  etc.)  developed  to  assist  student 

progress  through  the  lesson/s? 

122  42 

Yes  No 

144.  If  143  =  No,  why  not?  (Check  the  appropriate  responses.) 

Such  assistance  was  not  required. 

9  The  authoring  software  not  support  help  screens. 

Q  The  authoring  software  did  not  support  menus. 

_ 2  The  authoring  software  did  not  support  command  lines. 

8  Programmers  were  not  trained  to  ^velop  on-line  documentation. 

Other _ 


14S.  If  143  =  Yes,  what  type  of  help  was  offered?  (Check  the  appropriate  lesponse/s.) 

48  Tech  Older  (T.O.)  or  manual  references 
40  Diagrams 
71  Glossaries 
Lesson  mi^s 
.32.  Charts  or  tables 
.31  Sample  forms 

.2L  Other  _ 


146.  If  143  =  Yes,  how  was  the  on-line  help  presented?  ((^eck  the  appropriate  lesponse/s.) 

70  Hxed  menus  Qocated  at  particular  choice  points) 

29  "Pull-down"  mraus  (accessed  at  the  student’s  request) 

60  He4)  screens  (accessed  at  particular  points,  obscuring  lesson  content) 

6  Adjustable  help  screens  (accessed  at  the  student’s  request,  movable  tm  die  screen) 
22  Extended  help  screens  (accessed  at  the  student’s  request,  scrollable) 

36  Hxed  command  lines  (static  throughout  the  lesstm) 

22  Modified  command  lines  (reflecting  student  cations  as  they  change  through  the 
lesson) 

_16  Other _ _ _ 


147.  Was  an  on-line  review  conducted? 
148  13 

Yes  No 
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148.  If  147  s  No,  why  not?  (Check  the  appropriate  response/s.) 

_ 3  Review  was  not  required  by  the  SOW. 

_ I  There  was  not  sufficient  time  for  on-line  review. 

_ 9  (Qualified  SMEs  and/or  instiuct(»s  were  not  available. 

_ Q  On-line  review  was  not  budgeted. 

_ Z  Other  _ 


149.  If  147  s  Yes,  who  participated  in  the  review?  ((Theck  the  appropriate  response/s.) 

Program  Manager 

_ §  Contract  Manager 

_8l  Instructional  Developer 
Instructor  (End-user) 

_88  Subject  Matter  Expert 
48  Programmer 
Graphic  Artist 

2A  Other _ 


150.  If  147  s  Yes,  were  revisions  required? 

142  6 

Yes  No 

151.  If  150  s  Yes,  what  was  the  nature  of  the  required  revisions?  (Check  the  ippropriate 
response/s.) 

Objective  development 
Sequence  of  instruction 
_2S  Content 
Format 

JJJ  Orammatical/spelling^unctuation  errors 
_2P  Graphics 
Interactivity 

_3P  Advance  oiganizen  (titles,  bolding,  etc.) 

_69  Blanching 
_3?  Menus 

_L6  On-line  documentation  (help,  ccnnmand  lines) 

_14  Other  _ 
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152.  Was  lesson  production  completed  on  time? 
85  69 

Yes  No 


153.  If  152  s  No,  why  not?  (Check  the  t^propriate  response/s.) 

38  Pnsonnel  turnover  caused  delays. 

Time  to  learn  the  system  was  not  factored  in  to  the  schedule. 
On-line  review  took  longer  than  expected. 

38  Revisions  took  longer  than  expected 

21  Training  requirements  changed 

1ft  Design  requirements  changed 

_ Z.  CMI  requirements  changed 

n  Equipment  failures  caused  delays. 

16  Software  failures  caused  delays. 

21  Other  _ 


154.  Were  audit  trail  procedures  used  to  track  on-line  development  and  revisions? 
108  47 

Yes  No 


155.  Did  lesson  development  costs  exceed  the  budget? 
17  107 

Yes  No 


156.  If  155  s  Yes,  by  what  percentage? 


Mean  “ 

45Z 

Median  * 

25X 

Mode  = 

lOZ 

157.  If  155  is  Yes,  why?  (Check  the  qipropriate  response/s.) 


JA  More  time  was  required 

_ 6  Additional  personnel  were  required 

_ 1  Additional  hardware  was  required 

_ 5  Additional  software  was  required 

_ 9  I^rsotmel  turnover  occurred 

_ 3  Hardware  maintenance  was  necessary. 

_ 4  Software  maintenance  was  necessary. 

_ I  Training  requirements  changed 

_ 6  Design  requirements  changed 

3 
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D.  Validation.  The  purpose  of  validation  is  to  determine  whether  or  not  the  courseware 
teaches  the  objectives  effectively. 

158.  Have  you  been  involved  in  the  validation  of  CBT? 

114  121 

Yes  No 

(If  158  =  No,  please  go  on  to  Section  E,  (^esdon  167.) 


159.  If  158  =  Yes,  what  was  the  nature  of  your  involvement?  Check  the  appropriate 
response/s. 

Program  Manager 
19  System  Administrator 
_36  Instructor 
1^  Instructor  Supervisor 
Subject  Matter  Expert 
Computer  Systems  Expert 
27  Interviewer 
_2^  Analyst 

_18.  Other _ _ 


160.  If  158  =  Yes,  what  was  the  nature  of  ihe  validation?  (Check  the  appropriate  response/s.) 

_Z1  Observation  and  survey/interview  of  instructors  or  SMEs  taking  individual  lessons 
_§1.  Obsovation  and  survey/interview  of  individual  students  taking  individual  lessons 
_6L  Observation  and  surveyAnterview  of  small  groups  of  students  taking  individual 
lessons  or  groups  of  lessons 

_i£  Obsovation  ai^  surveyAnterview  of  large  groups  (usually  complete  classes)  of 
students  taking  the  total  course 

Observation  and  analyss  of  the  on-the-job  perfotmance  of  individuals  who  have 
taken  the  lesson/s 

Supervisor  evaluation  of  the  on-the-job  performance  of  individuals  who  have  taken 
the  lesson/s 

_12  Otho:  _ 


161.  Was  a  validation  plan  used? 
77  35 

Yes  No 
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162.  If  161  s  Yes,  what  was  covered  by  the  plan?  (Check  the  appropriate  response/s.) 

58  Procedures  for  student  involvement 

48  Procedures  for  instructor  involvement 

59  Data  collection  requirements 

47  Procedures  for  data  collection 

32  Procedures  for  data  analysis 

36  Data  reporting  requirements 

31  Procedures  for  reporting 

43  Evaluation  standt^ 

6  Other  _ _ 


163.  If  161  =  Yes,  was  the  validation  conducted  according  to  the  plan? 
72  5 

Yes  No 


164.  If  163  s  No,  why  not?  (Check  the  appropriate  response/s.) 

_ 1  The  plan’s  guidelines  were  not  required  by  the  SOW. 

_ 0.  Regulations  superseded  the  plan. 

_ 5  There  was  not  enough  time  to  follow  the  plan. 

_ 7  Necessary  personnel  were  unavailable. 

_ 3  Students  were  not  available. 

_ 5  OthCT;  _ 


16S.  Did  the  courseware  validate? 
94  7 

Yes  No 


166.  If  165  s  No,  why  not?  (Check  the  apinopriate  tesponse/s.) 

_ t  SttKlents  failed  to  meet  lesson  objectives. 

_ 5  Key  inaccuracies  in  content  woe  discovered. 

_ 3  It  was  discovered  that  the  courseware  did  not  meet  objective  requirements. 

_ 1  Students  were  not  able  to  apply  instruction  to  the  job  (in  the  field). 

_ 4  Other  _ 
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E.  Implementation.  This  involves  the  actual  use  of  CBT  to  provide  required  instruction  to 
students. 

167.  Have  you  been  involved  in  the  implementation  of  CBT? 

137  98 

Yes  No 

(If  167  s  No,  please  go  on  to  Section  F,  Question  206.) 

168.  If  167  s  Yes,  what  was  the  nature  of  your  involvement?  (Check  the  appropriate 
response/s.) 

Program  Manager 
System  Administrator 
^  Instructor 
HE  Instructor  Supervisor 

(Computer  Systems  Expat 

27  Other _ 


169.  Where  was  your  CBT  courseware  implemented?  (Check  the  appropriate  response/s.) 

.12.  Formal  training  enviromnent  (ie.,  schoolhouse) 

45  Informal  environment  (e.g.,  during  available  time  while  on-the-job) 

_ 7  Other _ 


170.  Was  the  CBT  optional  or  required  for  students? 

Optional 
_88  Required 

_ i  Other _ 


171.  Were  you  able  to  begin  implementing  CBT  on  time? 

79  31 

Yes  No 
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172.  If  171  =  No,  why  not?  (Check  the  appropriate  rcsponse/s.) 

20  Development  took  longer  than  expected 
10  Courseware  required  more  revisions  than  expected. 

_ 9  Hardware  not  available  at  training  sites. 

_ 1  Software  licenses  not  available  at  training  sites. 

12  Other _ 


173.  Was  a  system  administrator  (person  responsible  for  maintaining  and  troubleshooting 
hardware  and  authoring  software)  available  to  assist  with  system  operation? 

103  30 

Yes  No 


174.  If  173  =  Yes,  was  training  provided  for  the  system  administrator? 
81  17 

Yes  No 


175.  If  174  s  Yes,  what  was  the  administrator  trained  to  do?  (Check  the  appropriate 
response/s.) 

46  Install  hardware 

65  Install  software 

42  Register  instructors 

ftn  Register  students 

Perform  archival  functions 
65  Troubleshoot  the  system 

ih  Other  _ 


176.  If  173  s  No,  who  was  responsible  for  die  CBT  system  maintenance? 

_ 7  Program  Manager 

_ 4  Instructor 

_ 2  Designer/Dcvelopcr 

_ §  Programmer 

_ 2  Computer  Systems  Expert 

14  Other _ 
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177.  Were  instructors  available  to  assist  students  as  they  went  through  the  CBT  courseware? 
82  20 
Yes  No 


(If  177  =  No,  go  on  to  Question  190) 


178.  Were  any  instructors  reluctant  to  teach  using  CBT? 

67  37 

Yes  No 

179.  If  178  s  Yes,  what  reasons  were  given? 

Concern  about  student  management  (individualization,  varied  completion  times, 
etc.) 

Lack  of  familiari^  with  the  system 

_3l  C^oncem  about  system  reliabiUQr  (answer  judging,  student  advancement,  etc.) 

A9  Concern  about  the  effectiveness  of  the  CBT 
37  Concern  about  job  security 

Concern  about  instructor  roles  with  CBT 
11  Other  _ 


180.  Did  instructors  receive  training  on  the  system? 

90  14 

Yes  No 


181.  If  180  s  Yes,  what  training  was  provided?  (Check  die  appropriate  response/s.) 

_ 7  Install  hardware 

_24  Install  software 

_11  Register  instructors 

_5p  Register  students 

16  Perform  archival  functions 

_18  Troubleshoot  the  system 

Other  _ 
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182.  If  180  =  No,  why  not? 


_ 7  Training  was  not  required. 

_ 2  Training  was  not  available. 

_ 1  Training  on  the  system  was  not  budgeted  for  the  instmctors. 

_ 2  Other _ 


183.  Did  instructors  receive  training  in  CBT  instructional  techniques? 
57  44 

Yes  No 


184.  If  183  s  No,  why  was  no  training  offered  for  CBT  instructors? 

18  Training  was  not  required. 

10  Training  was  not  available. 

_ 7  Training  in  CBT  instructional  techniques  was  not  budgeted  for  instructors. 

_ 7  Otho: _ 


185.  If  183  =  Yes,  what  training  was  provided?  (Check  the  appropriate  response/s.) 

_18  Instructor  roles  with  C3T 
5n  System  operation  Gog  on,  log  off,  menu  use,  etc.) 

58  Student  data  management  (student  registratitHi,  data  collection,  reports,  etc.) 
_13  Course  structure 

Student  time  management 
Integration  of  CBT  with  traditional  instruction 
_ 3  Other  _ 


186.  What  was  the  student-instructor  ratio? 


12,  1 

To;T 

8  :  1 


Mean 

Median 

Mode 


187.  Was  the  student-instructor  ratio  sufficient  in  most  cases? 
76  5 

Yes  No 


188.  Was  the  instructor  able  to  meet  students*  needs  for  assistance? 

95  4 

Yes  No 
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189.  If  188  =  No,  why  not?  (Check  the  appropriate  response/s.) 

_ 2  The  student-teacher  ratio  exceeded  optimum  limits. 

_ P  There  was  no  opportunity  for  die  instructor  to  interact  with  the  students. 

_ 3  There  was  not  enough  time  fm*  the  instructor  to  fulfill  students’  requests  for 

assistance. 

_ P  The  instructor  was  not  familiar  with  material  covered  in  the  request 

_ 1  The  instructor  was  not  familiar  with  the  system  elements  being  questioned. 

_ p  Othen  _ 


190.  Were  any  students  reluctant  to  use  the  CBT? 
59  68 

Yes  No 


191.  If  190  =  Yes.  what  reasons  were  given?  (Check  the  appropriate  response/s.) 

A4  Lack  of  familiarity  with  the  system 
.28,  Lack  of  keyboarding  skills 

.12.  DifficulQr  reading  information  presented  on  the  computer  screen 
12  Concern  about  system  reliability  (answer  judging,  student  advancement,  etc.) 
22  Other  _ 


192.  Did  the  software  collect  all  required  student  data? 
86  25  134 

Yes  No  N/A 


193.  If  192  s  No,  what  could  not  be  collected?  (Check  the  apprtqniate  response/s.) 

_ 1  Student  data  (name,  job  title,  etc.) 

_ 6  Test  scores 

_ I  Time  in  lesson 

Embedded  question  scores 
Ji.  Number  of  attempts  per  question 

14  Answer  analysis 

10  Other  _ 


194.  Did  the  software  provide  hard  copy  reports  of  the  data  collected? 

94  15  136 

Yes  No  N/A 
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195.  Did  the  courseware  meet  perceived  instrucdonal  needs? 
107  16 

Yes  No 


196.  If  195  =  No,  what  reasons  were  given?  (Check  the  appropriate  response/s.) 

^  Individualization  disrupted  the  normal  schedule. 

3  Content  was  inaccurate. 

0  Objectives  were  not  met 

__i_  Objectives  did  not  accurately  meet  training  requirements. 

5  Lessons  were  pooiiy  presented. 

3  Lessons  were  loo  difficult 

^  Lessons  were  too  easy. 

6  Lessons  were  too  lengthy. 

_L  Lessons  were  incomplete. 

1  Lesson  sequence  was  confusing. 

6  Other.  _ 

197.  Did  students  "get  lost"  in  the  lesson/s? 

29  98 

Yes  No 

198.  If  197  =  Yes,  what  reasons  were  given?  (Check  the  appropriate  response/s.) 

10  Branching  was  confusing  or  didn't  wtnk. 

6  Menus  were  confusing  or  didn’t  woik. 

1  Titles  (and  subtitles)  were  confusing. 

_JL  There  were  too  many  layers  or  levels  in  the  lesson. 
a  Function  key  or  commit  line  options  did  not  return  the  student  to  the  lesson 
where  he  had  left  it 

5  Instruction  required  that  the  student  refCT  to  other  documents  or  parts  of  the 
lesson,  resulting  in  confusion. 

7  Other  _  - 


199.  Did  students  advance  through  the  lessons  as  planned? 
120  9 

Yes  No 
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200.  If  199  =  No,  why  not?  (Check  the  appropriate  response/s.) 

Students  found  ways  to  force  the  system  to  allow  them  to  advance. 

Test  scores  were  inaccurately  recorded. 

Branching  did  not  work  properly. 

Students  got  caught  in  an  "endless  loop." 

Other  _ 

201.  Did  the  students  use  the  on-line  help  (menus,  command  lines,  glossaries,  etc.)? 

95  24 

Yes  No 

202.  If  201  =  No,  what  reasons  were  given?  (Check  the  appropriate  response/s.) 

6  The  courseware  taught  the  material  well  enough. 

1  The  help  tvas  difficult  to  access. 

10  Students  preferred  to  get  assistance  from  the  instructor. 

11  Other  _ 

203.  Were  directions  supplied  to  assist  the  student  in  answering  the  questions? 

124  5 

Yes  No 


204.  Were  students  able  to  follow  the  directions? 

120  4 

Yes  No 


205.  If  204  s  No,  why  not?  ((Theck  the  appropriate  response/s.) 

0 
1 
0 
2 
2 
2 


Directions  were  inconsistent  in  wording. 
Directions  were  not  consistently  given. 
IKiections  were  confusing. 

Directions  were  too  lengthy. 

Directions  were  incomplete. 

Other  _ 
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F.  Maintenance.  This  relates  to  maintaining  and  troubleshooting  hardware  and  authoring 
software. 

206.  Have  you  been  involved  in  the  maintenance  of  CBT? 

92  102 

Yes  No 

(If  206  =  No,  please  go  to  Section  G.) 


207.  If  206  s  Yes,  what  was  the  nature  of  your  involvement?  (Check  the  appropriate 
response/s.) 

Maintaining  hardware 
52  Mainting  authoring  software 

32  Other: _ 


208.  Was  the  system  hardware  shared  in  any  way  (i.e.,  was  a  networked  system  used)? 
52  37 

Yes  No 

(If  208  s  No,  please  go  to  (^stion  213.) 


209.  Did  the  student  load  exceed  the  system*s  capabilities? 
21  56 

Yes  No 


210.  If  209  «  Yes,  what  was  the  solution?  (Check  the  iqrpropriate  lesponse/s.) 

7  Reschedule  classes 

_ 3  Reschedule  individual  students 

5  Require  that  students  "double  up" 

_ 6  Purchase  and  install  additional  student  stations 

10  Other  _ 


21 1.  Did  requirements  to  share  the  system  exceed  its  c^abilities? 
19  55 

Yes  No 
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212.  If  211  =  Yes,  what  was  the  nature  of  the  problem?  (Check  the  appropriate  response/s.) 

_ 1  Not  able  to  add  more  courseware 

_ i  Not  able  to  add  more  student  stations 

_ L  Not  able  to  add  applications  software  (word  processing,  databases,  etc.) 

_ i  Not  enough  merntny 

_li  Not  enough  speed 
_ 5  Not  enough  storage 

_ 4  Other _ _ 

213.  Did  you  experience  any  hat  iware  failures? 

82  29 

Yis  No 


214.  If  213  =  Yes,  what  happened?  (Check  the  appropriate  response/s.) 

A  single  station  failed. 

Multiple  stations  failed. 

Data  was  lost 

.26  Data  was  contaminated. 

34  The  system  would  not  start  up. 

_1I  Other  _ _ _ 


21S.  Were  hardware  maintenance  contracts  purchased? 
73  32 

Yes  No 


216.  If  215  s  Yes,  were  the  contracts  utilized? 
61  13 

Yes  No 


217.  If  216  =  Yes,  what  was  the  nature  of  the  problem?  (Check  the  appropriate  response/s.) 

System  failure 

System  expansion  desired 

"Bugs"  discovered  in  the  system 

Other  _ _ 
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218.  Did  you  experience  any  problems  with  the  authoring  software? 
51  45 

Yes  No 


219.  If  218  =  Yes,  what  types  of  problems  occurred?  (Check  the  appropriate  response/s.) 

10  Function  keys  did  not  woik. 

10  Menus  did  not  work. 

_15  Commands  (review,  return,  etc.)  did  not  work. 

1?  Branching  failed. 

_L3  Graphics  "bled"  into  text  or  othw  graphics. 

Lessons  *liung  up". 

20  (Men  _ 


220.  Were  authoring  software  maintenance  contracts  purchased? 
40  61 

Yes  No 


221.  Was  the  software  manufacturer’s  support  used? 
83  28 

Yes  No 


222.  If  221  =  Yes,  was  the  manufacturer  resporisive  in  meeting  the  needs  of  the  development 
staff? 

52  9 

Yes  No 


223.  If  221  s  No,  why  not?  (Check  the  t^tpropriate  response/s.) 

13  Support  was  not  needed. 

_ §  Support  was  unavailable. 

_ $  Required  procedures  for  obtaining  support  were  time-consuming. 

_ ^  A  support  contract  was  not  obtained. 

_ 5  Other  _ 


224.  Was  the  software  documentation  used? 

97  14 

Yes  No 
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225.  If  224  =  Yes,  was  the  software  documentation  sufficient  to  meet  the  needs  of  the 
development  team? 

53  19 

Yes  No 


226.  If  224  =  No,  why  not?  (Check  the  appropriate  rcsponse/s.) 

_ ^  The  documentation  was  not  available. 

_ 2  The  documentation  was  incomplete. 

_ 8  The  documentation  was  inaccurate. 

_ 3  The  documentation  was  out-of-date. 

_ 1  The  documentation  was  not  indexed  properiy. 

_ 1  Software  training  covered  the  documentation. 

_ 9  Other  _ 
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G.  DISCUSSION,  The  purpose  of  this  section  is  to  provide  you  with  the  opportunity  to 
suggest  areas  of  concern  or  interest  which  were  not  addressed  in  this  questionnaire.  Please  note 
your  comments  or  questions  below.  Thank  you  for  your  time. 
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